Wenhao Chai on FrontierCS 2

I’m Wenhao Chai, a second-year PhD student in Computer Science at Princeton, advised by Karthik Narasimhan. Before Princeton I did my master’s at the University of Washington and my bachelor’s at Zhejiang University, and I have spent time at Google DeepMind as a student researcher, at Pika Labs and at Microsoft Research Asia.

A lot of my work is about measuring what models can really do: I led MovieChat, one of the first models and benchmarks for hour-long video understanding, and I co-lead LiveCodeBench Pro. These days I mostly work on recursive self-improvement, which is a big part of why I care about evaluating models on real research.

FrontierCS 2 asks whether LLM agents can make contributions to frontier computer science R&D. It has 48 open-ended tasks in 13 domains, from machine-learning systems and high-performance computing to databases, security and robotics.

We build each task from a published paper and the code its authors released. We delete the code that implements the contribution of the paper, and that deleted code becomes the human reference. The agent inherits the rest of the repository and has to rebuild the missing part so that it beats the reference on the workloads the paper reported.

Much of computer science research looks like this: you pick up a published system, change one component, and get judged against the original implementation. Building tasks this way also gives every task a bar that already runs in its own repository, on the workloads of the paper.

In FrontierCS, an agent gets a problem statement and writes one program, and an evaluator scores it once after the run ends. On FrontierCS, ALE-Bench and AtCoder heuristic contests, agents still stay below the top human experts. Real research rarely works that way. You work inside an existing codebase, test changes as you go, and compare against what the authors built.

Benchmarks that do place agents in a real environment grade against tests or a leaderboard, and several stay within machine learning. Those that span other fields grade against a rubric or a reimplementation of the method. The closest design to ours is MLS-Bench, where each of 140 machine-learning tasks asks the agent to improve one named component. ResearchGym also removes the method of a paper from its repository. We take these ideas beyond machine learning to 13 domains, and we use the removed code itself as the bar.

The agent and the judge hold different information and run in separate sandboxes. The agent holds the repository. The judge holds the evaluator, a held-out final suite and the human reference. The sandbox of the agent reaches no network except a driver, so the agent cannot download the code of the authors.

The agent has five hours and can submit a patch at any time. For each patch it gets back only a development-suite score and a count of workloads that meet each threshold, with no reference score and no raw evaluator output. At the end, the agent names one patch as final, and only that patch is graded on the held-out suite, which adds workloads the agent never saw.

A run passes only if it beats the reference on every metric and every held-out workload. We also keep a private set of more than 100 tasks.

Claude Opus 5.5 is the best agent and passes 58% of its runs. Every other agent passes fewer than half, and no run passes on 16 of the 48 tasks.

What stood out to us is that a higher score does not mean a pass: most runs whose final score beats the human reference still miss a threshold on some metric or workload. On FCI, our shared capability scale for agents and tasks, the median task sits at about the strongest agent.

The leader also changes with the budget. DeepSeek V4.1 Flash leads at the smallest cost caps, though its costs are lower bounds, and Opus 5.5 leads at the largest.

Reading the patches showed ideas our measures miss. On QuarterPIRCryptography · EUROCRYPT 2024QuarterPIRMake each private lookup send fewer bytes and run faster.7 of 12 runs beat the reference, Muse Wonder 1.0 used the two checksums that let a RAID-6 disk array rebuild two failed disks, and cut far more bytes per lookup than any other passing run.

We expect it to hold up for a while, for a few reasons. The tasks span a wider range of difficulty than the agents do. The median task sits at about the FCI of the strongest agent, and the difficulty of a task is the FCI at which an agent passes it half the time. So today the best agent passes a typical task only about half the time. No run passes on 16 tasks.

The pass rule is strict: a run must beat the code of the authors on every metric and on every held-out workload, and most runs that beat the reference on the final score still fail somewhere. Beyond the 48 public tasks, we keep a private set of more than 100 tasks, to be announced in the future.

Our case studies show what the pass and the final score miss. On BBQNetwork · NSDI 2024BBQGenerate the per-level logic of a hardware priority queue.3 of 12 runs beat the reference, the final score ranks the largest passing circuit first, because it ignores circuit size. On QuarterPIRCryptography · EUROCRYPT 2024QuarterPIRMake each private lookup send fewer bytes and run faster.7 of 12 runs beat the reference, neither measure separates the run that removed far more bytes per lookup from the other passing runs, and we found that idea only by reading patches. On SVG-EARMLSys · ECCV 2026SVG-EARRebuild sparse attention for a video diffusion model.No run beat the reference, a run that is faster than the reference on every clip still fails on image quality in two clips.

With unlimited resources, we would measure the quality of the idea itself: whether an agent finds a better design and what it costs elsewhere, reported next to the pass. We would also run each agent on each task many times. Today each agent runs each task once, and the FCI intervals separate only agents that sit far apart.

Q Labs’ NanoGPT Slowrun. It flips the usual speedrun: everyone trains on the same 100M tokens of FineWeb, compute is unlimited, and the lowest validation loss wins.

I like it because it asks the question I think will matter most as data runs out, how much a model can learn from a fixed amount of data, and it turns that question into an open leaderboard where anyone can contribute. Ideas that a speedrun filters out for being expensive, like heavy regularization, second-order optimizers, ensembling and many epochs, finally get a fair test. It may be better known by now, but I still think it is underrated as a research environment.