清华大学和CUHK研究人员开源 AKV，是KV缓存工程在LLM agents的服务系统。 agents不总结旧历史，而是将所选的KV缓存项放下，继续再用其余的。 3个模型和4个agent 基准测试的平均成本下降了27.7%。 平均任务分提高1.7分。 代码为Apache-2.0许可并基于SGLang.
TL; DR 数据
- 服务系统加政策图书馆，没有重量。 在GPT-OSS-120B(5.1B活性),MiniMax-M2.7(230B活性，10B活性)和MiroThinker-1.7-Mini(30B活性，3B活性)上被测试。
- 运行在：NVIDIA GPU通过SGLang. 纸张在BF16中使用了8个A800-80GB GPU. 快速起动服务于2个GPU上的GPT-OSS-120B (tensor system 2.
- 性能：平均成本和+1.7分平均分比阈值总和低27.7%。
- 最佳：GPT-OSS-120B 在SWE-bench验证：每任务成本20.0到9.7分，得分40.0到43.2.
- 最差：MiniMax-M2.7在浏览器Comp-Plus上得分58.9分，文字工程的62.3下有3.4分。
什么是缓存工程，与上下文工程有什么不同？
AKV是一个清华服务系统，它让LLM agents放下并重新定位自己的KV缓存。
背景工程修饰或总结文本历史。 编辑此前缀使 KV 状态每晚无效。 所保留的后缀必须重新进行预填。 因此，利用方式总结了迟到和激烈的情况，这可能会失去中回合的细节。
缓存工程使文本轨迹未受影响。 它只会改变 KV 条目的可见度。 留存的条目无需再计算即可再使用。 他们仍然可以携带从投放的信息。 KV缓存占API总成本的82.4%。
AKV是怎样工作的？
AKV定义了3个缓存操作。 附录为新令牌的标准更新。 丢弃从未来注意力中删除所选条目而不改变位置。 重置重写保留了位置0到r-1的密钥，使用RoPE旋转。 价值保持不变，所以没有隐藏状态被重算。 在拼接中，将Repos 融资 GPT-OSS-120B的浏览器Comp-Plus通过率从55%增加到62%.
难的部分正在服务。 等同符可以在不同操作历史下产生不同的缓存状态。 普通前缀匹配可能会重用错误状态。 AKV 扩展了 SGLang 的 radex 树并附有滴放和再定位节点。 它只在没有主动请求仍然需要时才释放出KV. 片段批量注意在1个预填通道中恢复被拆迁状态。
AKV API是什么样子的？
该接口为无国籍和OpenAI相容。 客户端发送完整消息历史和两个可选字段。 滴入消息映射要滴出的信件的触发消息索引。 重新定位后列出各种指数，然后将职位压缩。
缓存政策从触发器、选择器和重新定位规则中构建这些字段。 内建政策采用回合制，代币制或模式制。 一个缓存图让模型调用trop context本身。
AKV在agent 基准测试上的表现如何？.
主要政策是工具响应滚动下降，保持最新的12个工具响应。 标准基线总结为96K令牌(米罗廷克为192K. Text Engineering assume at 48K. 基准测试被浏览comp-Plus,GAIA-Text,SWE-bench 验证和xbench-DeepSearch-2510.
4个基准测试的平均通过率(%)和成本(每任务美元). 资料来源：文件表1。
AKV还用削减价格21.9%与文字工程。 准确性差距长期打开。 在以96K令牌及以上的浏览器Comp-Plus上，GPT-OSS-120B与AKV的分数为37%. 标准分数24%，文本工程分数32%，每个项目页。
费用是按公布的费率计算的API等同估计数。 被重新定位的代币按缓存-打击率收费，这是故意保守的选择。
AKV在服务基准测试时有多快？.
服务测试在80个长文SWE-bench验证任务上运行了GPT-OSS-120B. 在每种方法的峰值设定中，AKV提供1.18x输出吞吐量和1.53x输入吞吐量。 P95请求延迟速度快1.51x.
以第16个货币计算，标准号的P95 耐用达到 229.90秒对AKV的30.63秒。 KV压力将Standard推入了重复再预装。 按解码步骤，全程注意时间从7.082分降至2.849分。
在100个SWE-bench根据GPU预算验证的任务上，AKV达到了约47%的通过率。 标准达到40.7%，文本工程达到32.3%.
agents能否调取自己的缓存政策？.
MiniMax-M2.7改进了20个浏览Comp-Plus任务的政策，然后运行了100个被搁置的任务。 重复6政策按基线准确性将成本削减了17.8%。 抽象报告，在专家设计的政策之外节省了7.1%。
将缓存控制显示为工具也有助于准确性。 GPT-OSS-120B在100任务子集上由52.1升至61.0. MiniMax-M2.7的花费从5.4美分上升到5.9美分。
AKV 与其他 KV 缓存方法
主要数字来自每个项目自己的纸张，并使用不同的设置。 它们不具有直接可比性。
关键外卖
- AKV让agents放下并重新定位KV缓存，而不是重写历史。
- 平均成本下降27.7%，在3个模型中得分上升1.7分。
- Repos将浏览器Comp-Plus上的GPT-OSS-120B从55%升至62%.
- 在16号货币时，P95延迟从229.90秒降至30.63秒。
- 支持今天只覆盖了Qwen,GPT-OSS和MiniMax-M2文本模型。
请检查access-date=中的日期值 (帮助) 页面存档备份，存于互联网档案馆页面存档备份，存于互联网档案馆页面存档备份。 并加入我们的通讯。 等等，你在电报里吗？现在你也可以加入我们的电报。
需要和我们合作推广您的 GitHub Repo OR Hugging Face Page OR Product Release OR Webinar等吗？联系我们
Asif Razzaq是Marktechpost AI Media Inc.的首席执行官。 作为一个有远见的企业家和工程师，阿西夫致力于利用人工智能的潜力为社会造福。 他最近的努力是人工智能媒体平台的发布，该平台以深入报道机器学习和深层学习新闻而出名，这些新闻在技术上既可靠又为广大观众所容易理解。 该平台每月有超过200万次观看，表明其受观众欢迎。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Tsinghua University and CUHK researchers open-sourced AKV, a serving system for KV cache engineering in LLM agents. Instead of summarizing old history, agents drop selected KV cache entries and keep reusing the rest. Across 3 models and 4 agent benchmarks, average cost falls 27.7%. Average task score rises 1.7 points. The code is Apache-2.0 licensed and builds on SGLang.
TL;DR
- What it is: Serving system plus policy library, no weights. Tested on GPT-OSS-120B (5.1B active), MiniMax-M2.7 (230B total, 10B active) and MiroThinker-1.7-Mini (30B total, 3B active).
- Runs on: NVIDIA GPUs via SGLang. Paper used 8 A800-80GB GPUs in BF16. Quick start serves GPT-OSS-120B on 2 GPUs (tensor parallel 2).
- Performance: 27.7% lower average cost and +1.7 points average score versus threshold summarization.
- Best: GPT-OSS-120B on SWE-bench Verified: cost per task 20.0 to 9.7 cents, score 40.0 to 43.2.
- Worst: MiniMax-M2.7 on BrowseComp-Plus scores 58.9, 3.4 points below Text Engineering’s 62.3.
What is cache engineering, and how is it different from context engineering?
AKV is a Tsinghua serving system that lets LLM agents drop and reposition their own KV cache.
Context engineering trims or summarizes the text history. Editing that prefix invalidates every later KV state. The retained suffix must then be re-prefilled. Harnesses therefore summarize late and aggressively, which can lose details from middle rounds.
Cache engineering leaves the text trajectory untouched. It only changes which KV entries stay visible to attention. Retained entries are reused without recomputation. They can still carry information from dropped messages. In the research paper’s cost analysis, KV cache reads made up 82.4% of total API cost.
How does AKV work?
AKV defines 3 cache operations. Append is the standard update for new tokens. Drop removes selected entries from future attention without changing positions. Repos rewrites retained keys to positions 0 through r-1 using a RoPE rotation. Values stay unchanged, so no hidden states are recomputed. In ablations, adding Repos raised GPT-OSS-120B’s BrowseComp-Plus pass rate from 55% to 62%.
The hard part is serving. Identical tokens can produce different cache states under different operation histories. Plain prefix matching could reuse the wrong states. AKV extends SGLang’s radix tree with drop and reposition nodes. It frees dropped KV only when no active request still needs it. Segment-batch attention restores evicted states in 1 prefill pass.
What does the AKV API look like?
The interface is stateless and OpenAI-compatible. Clients send the full message history and two optional fields. drop_message maps a trigger message index to the messages to drop. reposition lists the indices after which positions get compacted.
A CachePolicy builds those fields from a trigger, a selector and a reposition rule. Built-in policies are turn-based, token-based or model-based. A CacheTool lets the model call drop_context itself.
How does AKV perform on agent benchmarks?
The main policy is a tool-response rolling drop that keeps the latest 12 tool responses. The Standard baseline summarizes at 96K tokens (192K for MiroThinker). Text Engineering summarizes at 48K. Benchmarks are BrowseComp-Plus, GAIA-Text, SWE-bench Verified and xbench-DeepSearch-2510.
Average pass rate (%) and cost (US cents per task) across 4 benchmarks. Source: Table 1 of the paper.
AKV also cuts cost 21.9% versus Text Engineering. The accuracy gap opens on long runs. On BrowseComp-Plus at 96K tokens and above, GPT-OSS-120B scores 37% with AKV. Standard scores 24% and Text Engineering 32%, per the project page.
Costs are API-equivalent estimates from published rates. Repositioned tokens are charged at the cache-hit rate, a deliberately conservative choice.
How fast is AKV in serving benchmarks?
Serving tests ran GPT-OSS-120B on 80 long-context SWE-bench Verified tasks. At each method’s peak setting, AKV delivers 1.18x output throughput and 1.53x input throughput. P95 request latency is 1.51x faster.
At concurrency 16, Standard’s P95 latency hits 229.90 seconds versus 30.63 for AKV. KV pressure pushes Standard into repeated re-prefill. Per decode step, full-attention time falls from 7.082 to 2.849 ms.
On 100 SWE-bench Verified tasks under a GPU budget, AKV reached about 47% pass rate. Standard reached 40.7% and Text Engineering 32.3%.
Can agents tune their own cache policy?
MiniMax-M2.7 refined policies on 20 BrowseComp-Plus tasks, then ran 100 held-out tasks. The iteration-6 policy cut cost 17.8% versus Standard at baseline accuracy. The abstract reports 7.1% savings beyond the expert-designed policy.
Exposing cache control as a tool also helped accuracy. GPT-OSS-120B rose from 52.1 to 61.0 on a 100-task subset. MiniMax-M2.7’s cost rose, though, from 5.4 to 5.9 cents.
AKV vs other KV cache approaches
The main numbers come from each project’s own paper and use different setups. They are not directly comparable.
Key Takeaways
- AKV lets agents drop and reposition KV cache instead of rewriting history.
- Average cost falls 27.7% and score rises 1.7 points across 3 models.
- Repos lifts GPT-OSS-120B on BrowseComp-Plus from 55% to 62%.
- At concurrency 16, P95 latency drops from 229.90 to 30.63 seconds.
- Support covers only Qwen, GPT-OSS and MiniMax-M2 text models today.
Check out the Paper, GitHub Repo and Project Page. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
Asif Razzaq is the CEO of Marktechpost AI Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.