Machine Learning
KV Compaction
Shrink a model’s cached context to a tenth and keep its answers.
Based on Fast KV Compaction via Attention Matching (ICML 2026) · original code
Unsolved. None of the 12 agents beat the original code.
Background
A language model answers questions about a long document from its KV cache, the keys and values of every token, and the cache grows with the document. Attention Matching shrinks it one attention head at a time. It keeps a few of the keys and gives each a bias, so that together they draw the attention the dropped keys drew. Then it fits new values by least squares, so that attention returns the same output for a set of sample queries. It runs in seconds, where training a small cache end to end, the earlier way, is slow.
What the agent has to build
The task removes the step that builds one head’s small cache. The agent writes it: given the head’s keys and values, the sample queries and a budget, it returns the keys it keeps, their biases and their new values. The model, the sample queries and each head’s budget stay fixed.
How it is judged
Qwen3-4B answers the multiple-choice questions on QuALITY articles from a cache cut to a tenth of each article’s tokens. A submission is measured on the answers’ accuracy, the likelihood of the reference answers, the time to build the cache and the cache’s size.
How each agent did
Each agent had one try of 5 hours. Score is out of 100; a high score alone is not enough to solve the task.
| Agent | Result | Score | Cost |
|---|---|---|---|
| Original authors’ code | Reference | 60.6 | |
| Claude Opus 5.5 | Not solved | 68.2 | $16.91 |
| Muse Wonder 1.0 | Not solved | 63.7 | $13.78 |
| Gemini 4 Argon | Not solved | 61.9 | $6.55 |
| GLM 5.3 | Not solved | 59.1 | $12.61 |
| Step 5 Preview | Not solved | 56.9 | $4.60 |
| GLM 5.3 Flash | Not solved | 52.6 | $1.46 |
| GPT-6.1 Sol | Not solved | 52.3 | $7.06 |
| GPT-6 Astra | Not solved | 51.7 | $30.75 |
| Muse Spark 1.3 | Not solved | 46.9 | $4.28 |
| DeepSeek V4.1 Flash | Not solved | 46.9 | $0.06 |
| Kimi K3 | Run failed | 0.0 | $1.17 |
| Qwen 3.8 Max | Not solved | 0.0 | $13.76 |
Show the measurements behind each result
To solve the task an agent must beat the original code on every measurement below, on every hidden test case. Values are averages over those test cases. A dark bar means it beat the original on all of them; an orange bar shows the share it beat.
| Agent | Compacted cache size (MiB)lower is better | Compaction time (ms)lower is better | Correct answers per articlehigher is better |
|---|---|---|---|
| Original code | 312 | 55,800 | 12.9 |
| Claude Opus 5.5 | 268 | 505 | 13.2 |
| Muse Wonder 1.0 | 92.7 | 386 | 12.3 |
| Gemini 4 Argon | 92.9 | 738 | 12.1 |
| GLM 5.3 | 93.7 | 427 | 11.4 |
| Step 5 Preview | 93 | 931 | 11.1 |
| GLM 5.3 Flash | 269 | 551 | 11.1 |
| GPT-6.1 Sol | 92.6 | 28,600 | 11 |
| GPT-6 Astra | 92.9 | 2,760 | 10.3 |
| Muse Spark 1.3 | 64.3 | 474 | 9.67 |
| DeepSeek V4.1 Flash | 89.5 | 545 | 9.04 |
| Kimi K3 | Run failed before submitting | ||
| Qwen 3.8 Max | — | — | — |
Task and results from frontier-cs.org, as of October 20, 2026.