Tsinghua open-sources AKV, letting AI agents prune their own KV cache

What the Tsinghua team released

Researchers in Tsinghua University's NICS lab have open-sourced AKV, a serving system that lets AI agents decide which parts of their own memory to forget while the model keeps running. The code is on GitHub under the Apache 2.0 licence and is described in a paper, Cache Engineering for Agents, submitted to arXiv on 20 October 2026 by Tianyu Fu, Yu Wang and seven co-authors. MarkTechPost covered the release on 26 October and describes it as a joint Tsinghua and CUHK project.

AKV is not a model. It is two pieces of software: an agent-side library that plans what to remove from the model's working memory, and a modified version of the SGLang inference engine that carries out those removals without redoing the expensive work of reading everything again. The authors call the idea cache engineering, to contrast it with the context engineering that most agent tools use today.

The headline claim is that a simple policy built on AKV cut the average cost of agent tasks by 27.7% and raised average benchmark scores by 1.7 points, compared with the usual approach of summarising the conversation once it gets long. Those numbers are the authors' own, measured on three open models and four agent benchmarks, and have not been independently reproduced.

Why long agent runs get expensive

When a language model reads text, it stores intermediate results for every token in a key-value cache, or KV cache, so it does not have to recompute them on the next step. For a chatbot this cache is modest. For an agent that calls tools hundreds of times, reading web pages, files and test output, it grows with every round, and each new step has to read the whole thing again. The paper says KV cache reads accounted for 82.4% of total API cost in its analysis.

The common fix is to edit the text history: trim old tool results or replace them with a summary once the conversation crosses a token threshold. AI Lookout has explained that approach in our guide to context compaction for agents. The catch, the authors argue, is that changing anything early in the history invalidates every cached result after it, so the model must re-read, or re-prefill, the part it kept. Harnesses therefore wait until the context is very large and then cut hard, which can throw away details from the middle of a long task.

The paper puts numbers on that dilemma. On 100 BrowseComp-Plus research tasks with OpenAI's open-weight GPT-OSS-120B, trimming to the latest 12 tool responses cost 1.85 times as much as keeping the full context, because of all the re-reading. Summarising at 48,000 tokens cut cost by about 7% but lowered accuracy by 5 percentage points.

How drop and reposition work

Cache engineering leaves the text history alone and changes only what the model can still attend to. AKV adds two operations to the normal append step that stores new tokens. Drop hides selected cache entries from future attention without moving anything else. Reposition, or repos in the paper, rewrites the position encoding of the remaining keys so they sit at positions 0 onward again, closing the gaps. Values are untouched and nothing is recomputed.

Because the kept entries were originally computed while the dropped material was still visible, they can carry traces of it. That is part of the authors' explanation for why accuracy can rise rather than fall. Reposition matters for a practical reason too: without it, positions keep growing until the model hits its maximum context length. In an ablation, adding reposition raised GPT-OSS-120B's BrowseComp-Plus pass rate from 55% to 62%.

The default policy in the experiments is deliberately simple. It leaves the model's own messages in place, keeps the KV entries of the latest 12 tool responses, and drops older tool output as new results arrive. The paper justifies the choice with attention measurements showing that older tokens, and tool responses in particular, receive less attention than the model's own messages.

The serving problem AKV had to solve

The harder part is serving. Standard inference engines reuse cache by matching prefixes: if two requests start with the same tokens, they share the stored results. With drop and reposition in play, the paper points out, the same tokens can produce different cache states depending on the order of operations, so prefix matching alone could hand a request the wrong state.

AKV extends SGLang's radix tree, the structure that tracks shared prefixes, with nodes for drop and reposition operations. It frees a dropped entry only when no running request still needs it, and a technique the authors call segment-batch attention rebuilds any evicted states in a single prefill pass. This sits in the same part of the stack that commercial products such as Tensormesh's enterprise KV cache platform target, though AKV is aimed at what the agent chooses to keep rather than where the cache is stored.

To the client, the interface stays stateless and OpenAI-compatible. An agent sends its full message history as usual, plus two optional fields: drop_message, which says which earlier messages to drop after a given message is processed, and reposition, which says when to compact positions. The server does the bookkeeping. Usage reports add counts for cached, dropped and repositioned tokens.

What the benchmarks show

The main comparison runs MiniMax-M2.7, GPT-OSS-120B and MiroThinker-1.7-Mini on BrowseComp-Plus, a text-only GAIA set, SWE-bench Verified and xbench-DeepSearch-2510. AKV's rolling drop is set against two summarisation baselines built on the Pi agent's prompts: Standard, which summarises at 96,000 tokens (192,000 for MiroThinker), and Text Engineering, which summarises at 48,000.

Averaged across the four benchmarks, AKV had the best score for all three models: 53.7 against 52.4 and 52.8 for MiniMax-M2.7, 53.1 against 51.0 and 48.3 for GPT-OSS-120B, and 45.8 against 44.1 and 37.1 for MiroThinker. Average cost per task fell to 4.4, 7.3 and 14.8 US cents from 5.0, 11.2 and 20.4 under Standard. The biggest single saving was GPT-OSS-120B on SWE-bench Verified, where cost per task halved from 20.0 to 9.7 cents while the score rose from 40.0 to 43.2.

It is not a clean sweep. On BrowseComp-Plus, MiniMax-M2.7 scored 58.9 with AKV, the same as Standard and below Text Engineering's 62.3, and GPT-OSS-120B dipped from 38.5 to 37.5 on xbench. The authors attribute the overall accuracy gain to long runs: on 100 BrowseComp-Plus tasks with GPT-OSS-120B, all three methods scored about the same below 48,000 tokens, and the gap opened only after the summarisation thresholds kicked in.

On serving speed, eight NVIDIA A800 GPUs ran GPT-OSS-120B on the 80 SWE-bench Verified tasks with the longest contexts. At each method's best concurrency, AKV delivered 1.18 times the output throughput and a 1.51 times faster 95th-percentile request latency than Standard. The team also let a model rewrite the cache policy itself; policies found that way cut cost a further 7.1% over the hand-designed one on held-out tasks.

How developers can use it

Running AKV means hosting a model yourself. The repository README launches the server with a command that serves GPT-OSS-120B across two GPUs, and drop and reposition are currently enabled only for Qwen-family models from Qwen through Qwen3, GPT-OSS and the MiniMax-M2 family. It needs Rust and Python 3.12 for installation. Hosted APIs from OpenAI, Anthropic or Google cannot use it, because those providers control their own caches.

Integration is meant to be a single step. A harness builds its request as normal, passes it through a CachePolicy object that adds the drop and reposition fields, and sends it to the AKV server. Policies are made of a trigger that decides when to drop, a selector that decides what, and a reposition rule, with built-in turn-based, token-based and model-based versions.

The model-based option turns memory management into a tool. A DropContextTool lets the agent call drop_context on results it no longer needs, and a demo in the repository has an agent follow a chain of delivery records while discarding old ones. In the paper this gave the highest BrowseComp-Plus pass rates, 61% for GPT-OSS-120B and 64% for MiniMax-M2.7, though MiniMax's cost rose slightly because it ran more turns. It is a different take on agent memory from benchmarks such as Letta's FrontierMemory, which score what an agent chooses to remember.

Limits and open questions

The cost figures are API-equivalent estimates calculated from published token prices, not bills from a real deployment. The authors say they charged repositioned tokens at the cache-hit rate as a conservative choice. Serving tests used one model on one hardware setup, A800 GPUs running BF16 weights because the chips lack native support for the MXFP4 format GPT-OSS ships in.

The paper's own limitations section is short. It lists better cache strategies, training agents to work under this scheme, and support for linear-attention models as future work. The project is young: the repository was created on 28 September and had 93 stars when AI Lookout checked it. For teams already running open models on self-managed engines such as llm-d or SGLang, it is a concrete, testable idea. For everyone else, it is a sign that the cost of long agent runs may be cut inside the inference engine rather than by writing better summaries.