FrontierCS 2
A Benchmark for Computer Science R&D
FrontierCS 2 evaluates AI agents on their ability to do CS research. Each task removes the key contribution of a paper from the code released with it. An agent passes a task only if its implementation beats the original one on all metrics of our hidden workloads.
frontier-cs.org ↗Paper ↗Code ↗
Leaderboard
Pass rate (beat reference) across 48 tasks. 16 tasks are unsolved by every agent.
| # | Agent | Pass rate | Avg. cost per task |
|---|---|---|---|
| 1 | Claude Opus 5.5Claude Code$19.98 / task | 58% 28/48 | $19.98 |
| 2 | GPT-6 AstraCodex CLI$16.44 / task | 44% 21/48 | $16.44 |
| 3 | Gemini 4 ArgonPi$10.44 / task | 40% 19/48 | $10.44 |
| 4 | GPT-6.1 SolCodex CLI$3.04 / task | 35% 17/48 | $3.04 |
| 5 | GLM 5.3ZCode$9.78 / task | 25% 12/48 | $9.78 |
| 6 | Muse Wonder 1.0Muse Code$12.93 / task | 21% 10/48 | $12.93 |
| 7 | Kimi K3Kimi Code$8.12 / task | 21% 10/48 | $8.12 |
| 8 | DeepSeek V4.1 FlashDeepSeek Harness$0.11 / task | 15% 7/48 | $0.11 |
| 9 | GLM 5.3 FlashZCode$1.42 / task | 13% 6/48 | $1.42 |
| 10 | Muse Spark 1.3Muse Code$8.58 / task | 13% 6/48 | $8.58 |
| 11 | Qwen 3.8 MaxQwen Code$7.70 / task | 13% 6/48 | $7.70 |
| 12 | Step 5 PreviewStep Code$7.68 / task | 8% 4/48 | $7.68 |
Pass rate against cost
Each dot is one agent: how often it beat the reference, against what one run costs on average.
Pass rate by field
Choose a field to compare agents. Bold marks the highest pass rate.Share of each field’s tasks an agent solved, with the number of tasks under each field. Darker is higher; bold is the best in a column.
| Agent | Pass rate |
|---|---|
| Claude Opus 5.5 | 67% |
| GPT-6 Astra | 33% |
| Gemini 4 Argon | 33% |
| GPT-6.1 Sol | 33% |
| GLM 5.3 | 17% |
| Muse Wonder 1.0 | 17% |
| Kimi K3 | 33% |
| DeepSeek V4.1 Flash | 17% |
| GLM 5.3 Flash | 0% |
| Muse Spark 1.3 | 0% |
| Qwen 3.8 Max | 0% |
| Step 5 Preview | 33% |
| Agent | All48 | MLSys6 | Arch.3 | HPC4 | Net.3 | Theory1 | Bio.1 | ML9 | PL4 | SE1 | Sec.2 | Crypto3 | DB9 | Robot.2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 58% | 67% | 67% | 75% | 67% | 100% | – | 44% | 50% | – | – | 67% | 89% | – |
| GPT-6 Astra | 44% | 33% | 67% | 75% | 67% | 100% | – | 22% | 25% | – | – | 67% | 67% | – |
| Gemini 4 Argon | 40% | 33% | 67% | 50% | 67% | – | – | 33% | 25% | – | – | 67% | 56% | – |
| GPT-6.1 Sol | 35% | 33% | 67% | 50% | 67% | – | – | 11% | 25% | – | – | 67% | 56% | – |
| GLM 5.3 | 25% | 17% | 67% | 25% | – | – | – | 11% | 50% | – | – | 67% | 33% | – |
| Muse Wonder 1.0 | 21% | 17% | 33% | 25% | – | – | – | 11% | 25% | – | – | 67% | 33% | – |
| Kimi K3 | 21% | 33% | 33% | 50% | – | – | – | 11% | 25% | – | – | 33% | 22% | – |
| DeepSeek V4.1 Flash | 15% | 17% | 33% | 50% | – | – | – | 11% | 25% | – | – | – | 11% | – |
| GLM 5.3 Flash | 13% | – | – | 50% | – | – | – | – | 25% | – | – | 33% | 22% | – |
| Muse Spark 1.3 | 13% | – | 33% | 25% | – | – | – | 11% | – | – | – | – | 33% | – |
| Qwen 3.8 Max | 13% | – | – | 25% | – | – | – | 11% | – | – | – | 33% | 33% | – |
| Step 5 Preview | 8% | 33% | – | 50% | – | – | – | – | – | – | – | – | – | – |
All 48 tasks
Hardest first. Open any task to see what the agent had to build and how each one did.
- KV CompactionMachine LearningShrink a model’s cached context to a tenth and keep its answers.None yet
- Structured LWECryptographySolve as many structured-LWE instances as possible.None yet
- EdlibBioinformaticsRecover exact alignment paths in linear memory, fast.None yet
- EggProgramming LanguagesRestore an e-graph’s invariants with deferred rebuilding.None yet
- Flow-OPDMachine LearningSample the student’s trajectories to distill a text-rendering image model.None yet
- RAGENMachine LearningTurn multi-turn agent trajectories into training batches.None yet
- RUNLTSArchitecturePredict conditional branches with fewer mistakes.None yet
- LRRSecurityRepairing Regex Vulnerabilities via Localization-Guided InstructionsNone yet
- SVG-EARMLSysRebuild sparse attention for a video diffusion model.None yet
- YannakakisDatabaseRewrite join trees into faster two-phase semijoin plans.None yet
- Representation Fréchet LossMachine LearningKeep population statistics differentiable for a Fréchet-distance loss.None yet
- ContinuumMLSysSpeed up vLLM serving for multi-turn coding and tool agents.None yet
- Diffusion PolicyRoboticsSample a robot’s actions from a diffusion policy.None yet
- MuJoCo PlaygroundRoboticsTrain a quadruped to follow joystick commands.None yet
- Nano World ModelMachine LearningSample future video frames from a world model.1 of 12 agents
- BABBLEProgramming LanguagesFind library functions shared across a corpus of programs.1 of 12 agents
- SpectrumDatabaseMake EVM stack checkpoints cheap to create and restore.1 of 12 agents
- BBQNetworkGenerate the per-level logic of a hardware priority queue.3 of 12 agents
- MetaOptNetworkFind traffic matrices on which a combined heuristic does worst.3 of 12 agents
- LeJEPAMachine LearningImplement the loss that recovers a world model’s latents.2 of 12 agents
- FlashLib KNNMLSysMake brute-force GPU k-NN as fast as possible.1 of 12 agents
- DyTMachine LearningReplace a Transformer’s normalization layers with a learned tanh.2 of 12 agents
- ProgentSecurityProgent: Securing AI Agents with Privilege ControlNone yet
- CRESTSoftware EngineeringWrite the search strategy that picks which branch concolic testing flips next.None yet
- DeTokMachine LearningRestore the corruption training that makes an image tokenizer robust.1 of 12 agents
- RegionsSortHPCSort integers in place and in parallel.2 of 12 agents
- FARGO Query StoppingDatabaseDecide when an inner-product search can stop probing.4 of 12 agents
- PolygonProgramming LanguagesFind inputs on which two SQL queries disagree.1 of 12 agents
- SeerDatabasePredict the branches an EVM transaction will take.3 of 12 agents
- Binary Fuse FilterTheoryWrite the construction of a binary fuse filter.2 of 12 agents
- FlashLib DBSCANMLSysMake GPU DBSCAN as fast as possible.4 of 12 agents
- ShockwaveNetworkShare GPUs among ML jobs whose batch sizes keep changing.2 of 12 agents
- SCOPEDatabaseCount subgraph patterns for a batch of queries with shared plans.4 of 12 agents
- HygraHPCCompute core numbers of a hypergraph in parallel.5 of 12 agents
- Parallel DBSCANHPCCluster points with DBSCAN in parallel, exactly.4 of 12 agents
- ScaleGPMDatabasePick sampling or sparsification for each graph-pattern query.5 of 12 agents
- FlashLib K-MeansMLSysMake a GPU k-means step as fast as possible.5 of 12 agents
- KAMIArchitectureWrite the double-precision block matrix-multiply kernel for one GPU.6 of 12 agents
- CKKS NoiseCryptographyPredict the precision of CKKS computations, faster than the reference.8 of 12 agents
- QuarterPIRCryptographyMake each private lookup send fewer bytes and run faster.7 of 12 agents
- JoinBoostDatabaseTrain boosted trees over joined tables without joining them.7 of 12 agents
- GlensideProgramming LanguagesWrite the analysis that tracks tensor shapes through Glenside’s rewrites.9 of 12 agents
- FlashLib IVF-PQMLSysMake GPU IVF-PQ search as fast as possible.7 of 12 agents
- FARGO Query CoreDatabaseWrite the multi-probing query core of an inner-product search.8 of 12 agents
- Persistent TreapDatabaseApply concurrent updates to a persistent treap index.9 of 12 agents
- Clusterwise SpGEMMArchitectureSpeed up sparse matrix multiplication by clustering rows.8 of 12 agents
- Polar ExpressMachine LearningApproximate a matrix’s orthogonal factor in a few iterations.10 of 12 agents
- ParButterflyHPCCount butterflies in a bipartite graph in parallel.11 of 12 agents
Results from the FrontierCS 2 team at frontier-cs.org (MIT license), as of October 20, 2026.