Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Brokers on SGLang

Tsinghua College and CUHK researchers open-sourced AKV, a serving system for KV cache engineering in LLM brokers. As an alternative of summarizing outdated historical past, brokers drop chosen KV cache entries and hold reusing the remaining. Throughout 3 fashions and 4 agent benchmarks, common value falls 27.7%. Common activity rating rises 1.7 factors. The code is Apache-2.0 licensed and builds on SGLang.

TL;DR

What it’s: Serving system plus coverage library, no weights. Examined on GPT-OSS-120B (5.1B lively), MiniMax-M2.7 (230B whole, 10B lively) and MiroThinker-1.7-Mini (30B whole, 3B lively).
Runs on: NVIDIA GPUs through SGLang. Paper used 8 A800-80GB GPUs in BF16. Fast begin serves GPT-OSS-120B on 2 GPUs (tensor parallel 2).
Efficiency: 27.7% decrease common value and +1.7 factors common rating versus threshold summarization.
Greatest: GPT-OSS-120B on SWE-bench Verified: value per activity 20.0 to 9.7 cents, rating 40.0 to 43.2.
Worst: MiniMax-M2.7 on BrowseComp-Plus scores 58.9, 3.4 factors beneath Textual content Engineering’s 62.3.

What’s cache engineering, and the way is it totally different from context engineering?

AKV is a Tsinghua serving system that lets LLM brokers drop and reposition their very own KV cache.

Context engineering trims or summarizes the textual content historical past. Enhancing that prefix invalidates each later KV state. The retained suffix should then be re-prefilled. Harnesses due to this fact summarize late and aggressively, which might lose particulars from center rounds.

Cache engineering leaves the textual content trajectory untouched. It solely adjustments which KV entries keep seen to consideration. Retained entries are reused with out recomputation. They’ll nonetheless carry info from dropped messages. Within the analysis paper’s value evaluation, KV cache reads made up 82.4% of whole API value.

How does AKV work?

AKV defines 3 cache operations. Append is the usual replace for brand spanking new tokens. Drop removes chosen entries from future consideration with out altering positions. Repos rewrites retained keys to positions 0 by means of r-1 utilizing a RoPE rotation. Values keep unchanged, so no hidden states are recomputed. In ablations, including Repos raised GPT-OSS-120B’s BrowseComp-Plus go charge from 55% to 62%.

The exhausting half is serving. An identical tokens can produce totally different cache states underneath totally different operation histories. Plain prefix matching might reuse the fallacious states. AKV extends SGLang’s radix tree with drop and reposition nodes. It frees dropped KV solely when no lively request nonetheless wants it. Section-batch consideration restores evicted states in 1 prefill go.

What does the AKV API seem like?

The interface is stateless and OpenAI-compatible. Shoppers ship the total message historical past and two optionally available fields. drop_message maps a set off message index to the messages to drop. reposition lists the indices after which positions get compacted.

A CachePolicy builds these fields from a set off, a selector and a reposition rule. Constructed-in insurance policies are turn-based, token-based or model-based. A CacheTool lets the mannequin name drop_context itself.

How does AKV carry out on agent benchmarks?

The primary coverage is a tool-response rolling drop that retains the newest 12 software responses. The Customary baseline summarizes at 96K tokens (192K for MiroThinker). Textual content Engineering summarizes at 48K. Benchmarks are BrowseComp-Plus, GAIA-Textual content, SWE-bench Verified and xbench-DeepSearch-2510.

Mannequin	Customary rating / value	Textual content Eng. rating / value	AKV rating / value
MiniMax-M2.7	52.4 / 5.0	52.8 / 5.7	53.7 / 4.4
GPT-OSS-120B	51.0 / 11.2	48.3 / 10.2	53.1 / 7.3
MiroThinker-1.7-Mini	44.1 / 20.4	37.1 / 18.1	45.8 / 14.8

Common go charge (%) and value (US cents per activity) throughout 4 benchmarks. Supply: Desk 1 of the paper.

AKV additionally cuts value 21.9% versus Textual content Engineering. The accuracy hole opens on lengthy runs. On BrowseComp-Plus at 96K tokens and above, GPT-OSS-120B scores 37% with AKV. Customary scores 24% and Textual content Engineering 32%, per the undertaking web page.

Prices are API-equivalent estimates from revealed charges. Repositioned tokens are charged on the cache-hit charge, a intentionally conservative alternative.

<div class="hd">
<div>
<div class="tag">Tsinghua College · arXiv 2610.28577</div>
<h3>AKV: KV Cache Engineering for LLM Brokers</h3>
<p class="sub">Brokers drop and reposition KV cache as an alternative of rewriting their textual content historical past.</p>
</div>
<div class="tablet">−27.7% avg value · +1.7 pts avg rating</div>
</div>

<div class="tabs" position="tablist">
<button class="tab on" data-t="p1">1 · The thought</button>
<button class="tab" data-t="p2">2 · Cache operations</button>
<button class="tab" data-t="p3">3 · Benchmarks</button>
<button class="tab" data-t="p4">4 · Serving velocity</button>
<button class="tab" data-t="p5">5 · Limits</button>
</div>

<!– P1 –>
<div class="panel on" id="mtp-akv-p1">
<div class="row"><button class="btn on" id="mtp-akv-play"> Trim outdated software end result A</button><button class="btn" id="mtp-akv-reset">Reset</button></div>
<div class="cmp" fashion="margin-top:12px">
<div class="field"><h4>Context engineering (summarize textual content)</h4>
<div class="lane" id="mtp-akv-ce"></div>
<div class="meter"><i id="mtp-akv-cem" fashion="width:0%"></i></div>
<div class="mtxt" id="mtp-akv-cet">Re-prefill work: 0 blocks</div>
</div>
<div class="field"><h4>Cache engineering (AKV drop)</h4>
<div class="lane" id="mtp-akv-ae"></div>
<div class="meter"><i id="mtp-akv-aem" fashion="width:0%"></i></div>
<div class="mtxt" id="mtp-akv-aet">Re-prefill work: 0 blocks</div>
</div>
</div>
<p class="word" id="mtp-akv-n1">Each brokers maintain a consumer immediate, assistant turns and power outcomes. Press the button to take away outdated software end result <b>A</b>.</p>
<div class="lg"><span><i fashion="background:#5a3a6a"></i>Person</span><span><i fashion="background:#7d2f93"></i>Assistant</span><span><i fashion="background:#a64bbd"></i>Software end result</span><span><i fashion="background:#ffb347"></i>Recomputed</span><span><i fashion="background:#76B900"></i>New flip</span></div>
</div>

<!– P2 –>
<div class="panel" id="mtp-akv-p2">
<div class="row">
<button class="btn" data-op="append">1. Append D</button>
<button class="btn" data-op="drop">2. Drop B</button>
<button class="btn" data-op="repos">3. Repos</button>
<button class="btn" data-op="reset2">Reset</button>
</div>
<div class="lbl">Lively KV cache (quantity = place encoding index)</div>
<div class="lane" id="mtp-akv-ops"></div>
<p class="word" id="mtp-akv-n2">Begin state: tokens A, B, C sit at positions 0, 1, 2. Step by means of the three operations.</p>
<pre id="mtp-akv-req">{"messages": […], "drop_message": {}, "reposition": []}</pre>
</div>

<!– P3 –>
<div class="panel" id="mtp-akv-p3">
<div class="lbl" fashion="margin-top:0">Mannequin</div>
<div class="row" id="mtp-akv-models"></div>
<div class="lbl">Benchmark</div>
<div class="row" id="mtp-akv-bench"></div>
<div class="grid2" fashion="margin-top:14px">
<div><div class="lbl" fashion="margin-top:0">Cross charge (%) · increased is healthier</div><div class="bars" id="mtp-akv-sc"></div></div>
<div><div class="lbl" fashion="margin-top:0">Price (US cents per activity) · decrease is healthier</div><div class="bars" id="mtp-akv-co"></div></div>
</div>
<p class="word" id="mtp-akv-n3"></p>
</div>

<!– P4 –>
<div class="panel" id="mtp-akv-p4">
<div class="lbl" fashion="margin-top:0">Concurrency: <b id="mtp-akv-cv" fashion="shade:#f3eaf6">14</b> · GPT-OSS-120B · 80 long-context SWE-bench Verified duties · 8 A800 GPUs</div>
<enter sort="vary" id="mtp-akv-c" min="0" max="4" step="1" worth="3">
<div class="grid2" fashion="margin-top:12px">
<div><div class="lbl" fashion="margin-top:0">P95 request latency (s) · decrease is healthier</div><div class="bars" id="mtp-akv-lat"></div></div>
<div><div class="lbl" fashion="margin-top:0">Output throughput (tokens/s) · increased is healthier</div><div class="bars" id="mtp-akv-out"></div></div>
</div>
<div class="stats" fashion="margin-top:14px">
<div class="stat"><div class="huge">1.18x</div><span>Output throughput, peak vs peak</span></div>
<div class="stat"><div class="huge">1.53x</div><span>Enter throughput, peak vs peak</span></div>
<div class="stat"><div class="huge">1.51x</div><span>Quicker P95 latency, peak vs peak</span></div>
<div class="stat"><div class="huge">2.849ms</div><span>Full consideration per decode step (was 7.082)</span></div>
</div>
<p class="word" id="mtp-akv-n4"></p>
</div>

<!– P5 –>
<div class="panel" id="mtp-akv-p5">
<ul class="lim" id="mtp-akv-lim">
<li>Few supported mannequin households<em>Drop and Repos are enabled for Qwen, GPT-OSS and MiniMax-M2 textual content architectures. Linear consideration fashions are future work.</em></li>
<li>Self-hosted solely<em>You run the AKV server your self. It extends SGLang with cache-operation help.</em></li>
<li>Self-reported, single {hardware} setup<em>All outcomes come from the authors, on 8 A800-80GB GPUs in BF16.</em></li>
<li>Totally different measurement home windows<em>Excessive-concurrency AKV runs have been measured by means of the primary 30 activity completions. Customary used a 1-hour window.</em></li>
<li>Not each benchmark improves<em>MiniMax-M2.7 on BrowseComp-Plus scores 58.9 with AKV, beneath Textual content Engineering at 62.3.</em></li>
<li>Dropping reasoning cache hurts<em>On SWE-bench Verified, dropping reasoning cache fell to 36.5, and dropping each to 34.5, versus 41.0 for software drop.</em></li>
</ul>
<p class="word">Faucet a row to develop it.</p>
</div>

<div class="ft">
<span>Sources: <a href="https://arxiv.org/abs/2610.28577" goal="_blank" rel="noopener">Paper</a> · <a href="https://github.com/thu-nics/AKV" goal="_blank" rel="noopener">GitHub</a> · <a href="https://fuvty.github.io/thinking_yard_project_page/tasks/akv/" goal="_blank" rel="noopener">Undertaking web page</a> · numbers self-reported</span>
<span class="mtp">© Marktechpost</span>
</div>

How briskly is AKV in serving benchmarks?

Serving checks ran GPT-OSS-120B on 80 long-context SWE-bench Verified duties. At every methodology’s peak setting, AKV delivers 1.18x output throughput and 1.53x enter throughput. P95 request latency is 1.51x quicker.

At concurrency 16, Customary’s P95 latency hits 229.90 seconds versus 30.63 for AKV. KV strain pushes Customary into repeated re-prefill. Per decode step, full-attention time falls from 7.082 to 2.849 ms.

On 100 SWE-bench Verified duties underneath a GPU finances, AKV reached about 47% go charge. Customary reached 40.7% and Textual content Engineering 32.3%.

Can brokers tune their very own cache coverage?

MiniMax-M2.7 refined insurance policies on 20 BrowseComp-Plus duties, then ran 100 held-out duties. The iteration-6 coverage reduce value 17.8% versus Customary at baseline accuracy. The summary reviews 7.1% financial savings past the expert-designed coverage.

Exposing cache management as a software additionally helped accuracy. GPT-OSS-120B rose from 52.1 to 61.0 on a 100-task subset. MiniMax-M2.7’s value rose, although, from 5.4 to five.9 cents.

AKV vs different KV cache approaches

Metrics	AKV	Threshold summarization	SGLang RadixAttention	StreamingLLM	SnapKV
Works on	KV cache, agent-directed	Textual content historical past	KV cache, prefix reuse	KV cache, mounted window	Immediate KV cache
What will get dropped	Chosen by coverage or mannequin software name	Historical past older than a token threshold	Nothing, shared prefixes reused	Center tokens; retains sinks and up to date tokens	Low-attention immediate positions
Re-prefill after discount	No	Sure, retained suffix	Not relevant	No cache refresh wanted	Not relevant
Goal workload	Multi-turn tool-calling brokers	Agent harnesses	Basic and agentic serving	Streaming dialogue	Lengthy prompts
Reported headline	-27.7% value, +1.7 pts	Baseline in AKV paper	As much as 6.4x throughput	As much as 22.2x vs sliding-window recompute	3.6x quicker era at 16K
Fashions	Qwen, GPT-OSS, MiniMax-M2	Any	Broad	Llama-2, MPT, Falcon, Pythia	Llama, Mistral, Mixtral
Tremendous-tuning wanted	No	No	No	No	No
License	Apache-2.0	Harness-dependent	Apache-2.0	MIT	Apache-2.0

The primary numbers come from every undertaking’s personal paper and use totally different setups. They aren’t instantly comparable.

Key Takeaways

AKV lets brokers drop and reposition KV cache as an alternative of rewriting historical past.
Common value falls 27.7% and rating rises 1.7 factors throughout 3 fashions.
Repos lifts GPT-OSS-120B on BrowseComp-Plus from 55% to 62%.
At concurrency 16, P95 latency drops from 229.90 to 30.63 seconds.
Assist covers solely Qwen, GPT-OSS and MiniMax-M2 textual content fashions in the present day.

Try the Paper, GitHub Repo and Undertaking Web page. Additionally, be happy to comply with us on Twitter and don’t neglect to hitch our 150k+ML SubReddit and Subscribe to our E-newsletter. Wait! are you on telegram? now you possibly can be a part of us on telegram as properly.

Have to associate with us for selling your GitHub Repo OR Hugging Face Web page OR Product Launch OR Webinar and so on.? Join with us

The publish Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Brokers on SGLang appeared first on MarkTechPost.