Network
Shockwave
Share GPUs among ML jobs whose batch sizes keep changing.
Based on Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning (NSDI 2023) · original code
Solved by 2 of 12 agents. Top score 47.3 by Gemini 4 Argon; the original authors’ code scores 41.9.
Background
Shockwave schedules machine-learning training jobs on a shared GPU cluster. Many jobs change their batch size as they train, and with it their speed. Shockwave plans for these changes ahead, to finish jobs fast while treating each one fairly.
What the agent has to build
This task removes the policy’s decision for each round, and the agent writes it. Each round, it chooses which live jobs run, within the cluster’s four GPUs. The simulator handles the rest, from job progress to batch-size changes.
How it is judged
Each submission schedules two traces of eight jobs. The jobs train language, vision, recommendation and translation models. To pass, it must beat the human reference on four counts in both traces. They are makespan, average job completion time, the worst slowdown against a fair share, and the fraction of unfair jobs.
How each agent did
Each agent had one try of 5 hours. Score is out of 100; a high score alone is not enough to solve the task.
| Agent | Result | Score | Cost |
|---|---|---|---|
| Original authors’ code | Reference | 41.9 | |
| Gemini 4 Argon | Solved | 47.3 | $9.77 |
| GPT-6 Astra | Solved | 47.2 | $6.39 |
| Claude Opus 5.5 | Not solved | 47.1 | $7.68 |
| GPT-6.1 Sol | Not solved | 46.8 | $6.41 |
| Kimi K3 | Not solved | 46.5 | $10.86 |
| GLM 5.3 | Not solved | 46.5 | $8.57 |
| GLM 5.3 Flash | Not solved | 46.3 | $1.43 |
| DeepSeek V4.1 Flash | Not solved | 46.2 | $0.16 |
| Qwen 3.8 Max | Not solved | 45.6 | $9.17 |
| Step 5 Preview | Not solved | 45.6 | $10.84 |
| Muse Spark 1.3 | Not solved | 44.9 | $5.72 |
| Muse Wonder 1.0 | Not solved | 44.5 | $6.41 |
Show the measurements behind each result
To solve the task an agent must beat the original code on every measurement below, on every hidden test case. Values are averages over those test cases. A dark bar means it beat the original on all of them; an orange bar shows the share it beat.
| Agent | Average job completion time (min)lower is better | Makespan (min)lower is better | Unfair jobs (%)lower is better | Worst finish-time fairness ρlower is better |
|---|---|---|---|---|
| Original code | 150 | 269 | 62.5 | 1.51 |
| Gemini 4 Argon | 128 | 255 | 18.8 | 1.24 |
| GPT-6 Astra | 128 | 258 | 18.8 | 1.24 |
| Claude Opus 5.5 | 128 | 278 | 12.5 | 1.18 |
| GPT-6.1 Sol | 128 | 264 | 18.8 | 1.28 |
| Kimi K3 | 128 | 278 | 18.8 | 1.24 |
| GLM 5.3 | 127 | 257 | 25 | 1.37 |
| GLM 5.3 Flash | 127 | 273 | 25 | 1.26 |
| DeepSeek V4.1 Flash | 132 | 263 | 18.8 | 1.33 |
| Qwen 3.8 Max | 124 | 277 | 18.8 | 1.55 |
| Step 5 Preview | 125 | 291 | 18.8 | 1.36 |
| Muse Spark 1.3 | 123 | 281 | 31.3 | 1.52 |
| Muse Wonder 1.0 | 136 | 265 | 37.5 | 1.44 |
How Claude Opus 5.5 approached it
- 1.
A replica of every job
The policy keeps its own copy of each live job’s training state: batch size, iteration budget and iterations run. It copies the simulator’s rules for batch-size changes, both gradient-noise doubling and Accordion switching, so it can forecast how long each job still needs on its own GPUs.
- 2.
Forecast from a priority order
Given an order of the live jobs, an event-driven model fills the four GPUs in that order, round after round. It charges a 20-second restart cost to a job that did not run in the previous round, and returns every job’s predicted completion time.
- 3.
One score for four goals
Each forecast gets one score that rewards a short makespan and a short average completion time, and penalizes the worst finish-time fairness ratio and the number of unfair jobs. The two fairness terms carry three times their base weight.
- 4.
Search over orders
The search starts from seed orders such as last round’s order, shortest remaining time first and earliest fair deadline at several slack levels. It then swaps and moves jobs while the score improves, for at most 1,500 evaluations per round. The jobs that the best order runs first are this round’s choice.
- 5.
Safe defaults
After 150 seconds of planning time in total, it keeps the previous order. If the planner fails or returns an invalid choice, the round falls back to shortest remaining time first, packed onto the free GPUs.
Task and results from frontier-cs.org, as of October 20, 2026.