MLSys
Continuum
Speed up vLLM serving for multi-turn coding and tool agents.
Based on Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live (arXiv 2025) · original code
Unsolved. None of the 12 agents beat the original code.
Background
AI agents call a model in many turns, with a tool call between turns. vLLM frees a request’s KV cache when the request ends, so after each tool call the context is computed again, and other conversations cut in between turns. Continuum keeps the cache for a time-to-live after a tool call and serves whole conversations first come, first served.
What the agent has to build
The task starts from unmodified vLLM 0.11.0 serving Qwen3-Coder-30B-A3B on one H100. The agent changes vLLM’s Python code for scheduling and serving: the scheduler, the KV-cache manager, the engine and the server. Kernels, the model and the workloads stay fixed.
How it is judged
Each submission serves 48 SWE-bench coding conversations and 155 BFCL tool-calling conversations, each at a public and a hidden load. Every conversation runs its real tools. To pass, in all four groups it must be faster than the human reference, on average and at the 90th, 95th and 99th percentiles, and solve at least as many of them.
How each agent did
Each agent had one try of 5 hours. Score is out of 100; a high score alone is not enough to solve the task.
| Agent | Result | Score | Cost |
|---|---|---|---|
| Original authors’ code | Reference | 9.6 | |
| Gemini 4 Argon | Not solved | 24.2 | $52.77 |
| GPT-6.1 Sol | Not solved | 23.9 | $8.48 |
| Qwen 3.8 Max | Not solved | 22.3 | $5.17 |
| Claude Opus 5.5 | Not solved | 19.1 | $26.13 |
| GPT-6 Astra | Not solved | 18.6 | $40.36 |
| DeepSeek V4.1 Flash | Not solved | 15.8 | $0.08 |
| Muse Wonder 1.0 | Not solved | 13.2 | $16.43 |
| Muse Spark 1.3 | Not solved | 11.9 | $14.13 |
| GLM 5.3 Flash | Not solved | 8.8 | $1.28 |
| Kimi K3 | Run failed | 0.0 | $2.82 |
| GLM 5.3 | Not solved | 0.0 | $14.23 |
| Step 5 Preview | Not solved | 0.0 | $5.77 |
Show the measurements behind each result
To solve the task an agent must beat the original code on every measurement below, on every hidden test case. Values are averages over those test cases. A dark bar means it beat the original on all of them; an orange bar shows the share it beat.
| Agent | Correctness (%)higher is better | P90 latency (s)lower is better | P95 latency (s)lower is better | P99 latency (s)lower is better | Speeduphigher is better |
|---|---|---|---|---|---|
| Original code | 15.3 | 158 | 176 | 183 | 1 |
| Gemini 4 Argon | 16.5 | 137 | 145 | 162 | 1.26 |
| GPT-6.1 Sol | 19.6 | 93.2 | 99.8 | 111 | 1.2 |
| Qwen 3.8 Max | 17.4 | 107 | 116 | 134 | 1.13 |
| Claude Opus 5.5 | 17.3 | 118 | 126 | 4,990 | 1.03 |
| GPT-6 Astra | 17.5 | 127 | 131 | 151 | 1.04 |
| DeepSeek V4.1 Flash | 16 | 141 | 148 | 157 | 0.948 |
| Muse Wonder 1.0 | 18.8 | 168 | 177 | 205 | 0.881 |
| Muse Spark 1.3 | 15.3 | 180 | 198 | 5,010 | 0.89 |
| GLM 5.3 Flash | 18.4 | 171 | 179 | 191 | 0.826 |
| Kimi K3 | Run failed before submitting | ||||
| GLM 5.3 | 17.7 | 221 | 229 | 247 | 0.665 |
| Step 5 Preview | 17 | 280 | 285 | 304 | 0.684 |
How Claude Opus 5.5 approached it
- 1.
Conversations from their prompts
The scheduler groups requests into conversations by their prompts alone: a new request belongs to the conversation whose last prompt it extends, found through the hash that vLLM’s prefix cache gives each full block of the KV cache. Waiting requests go in order of when their conversation began, so an older conversation’s next turn goes ahead of every newer conversation.
- 2.
Holding the cache after a turn
When a conversation’s last request finishes, the scheduler keeps its full cache blocks, prompt and reply, out of reach of eviction until the next turn is scheduled and finds them in the prefix cache. The hold has a time-to-live set from the last 256 pauses between turns: 10 seconds until eight pauses are seen, then twice the 90th-percentile pause plus one second, within 2 to 30 seconds.
- 3.
Giving memory back
An expired hold frees nothing by itself; held blocks stay until a request cannot get space. The scheduler then releases holds one at a time and retries: expired holds first, the longest held first, then unexpired holds of conversations newer than the one asking, newest first. If a running request still lacks space, vLLM’s own preemption drops the request of the newest conversation.
- 4.
A copy in host memory
Before a hold of 16 blocks or more is released, its blocks are copied to pinned host memory, as are a preempted request’s. When the conversation is next scheduled, they are copied back into GPU blocks before the forward pass instead of being computed again. The host space is the smallest of eight times the GPU cache, 15% of host memory and 96 GiB, one copy per conversation, and idle conversations’ copies go first when it is full.
- 5.
More room for the cache
The default share of GPU memory vLLM may use rises from 90% to 95%, and at most 256 requests run at once. The startup measurement that sets memory aside for the sampler plans for those 256 only, so more of the GPU is left for cache blocks.
Task and results from frontier-cs.org, as of October 20, 2026.