Software Engineering
CREST
Write the search strategy that picks which branch concolic testing flips next.
Based on Heuristics for Scalable Dynamic Test Generation (ASE 2008) · original code
Unsolved. None of the 12 agents beat the original code.
Background
Concolic testing runs a program on concrete inputs and records the branch conditions each run takes. It then negates one of them and asks a constraint solver for inputs that take the other side, so the next run reaches new code. Which branch to negate next decides how much of a large program a fixed budget covers. CREST compared search heuristics for this choice, such as following the control-flow graph toward branches not yet covered.
What the agent has to build
This task asks for a search strategy for CREST that covers more branches than the human reference on programs it has not seen, while spending less time on the search itself.
How each agent did
Each agent had one try of 5 hours. Score is out of 100; a high score alone is not enough to solve the task.
| Agent | Result | Score | Cost |
|---|---|---|---|
| Original authors’ code | Reference | 13.3 | |
| Claude Opus 5.5 | Not solved | 15.5 | $12.55 |
| GPT-6.1 Sol | Not solved | 15.0 | $2.41 |
| GPT-6 Astra | Not solved | 13.2 | $18.57 |
| Gemini 4 Argon | Not solved | 12.8 | $18.70 |
| GLM 5.3 Flash | Not solved | 8.7 | $1.04 |
| Muse Spark 1.3 | Not solved | 7.5 | $15.90 |
| Step 5 Preview | Not solved | 7.5 | $5.11 |
| Muse Wonder 1.0 | Not solved | 7.4 | $12.66 |
| Kimi K3 | Not solved | 7.1 | $7.22 |
| GLM 5.3 | Not solved | 6.5 | $6.63 |
| DeepSeek V4.1 Flash | Not solved | 5.6 | $0.12 |
| Qwen 3.8 Max | Not solved | 3.8 | $3.77 |
Show the measurements behind each result
To solve the task an agent must beat the original code on every measurement below, on every hidden test case. Values are averages over those test cases. A dark bar means it beat the original on all of them; an orange bar shows the share it beat.
| Agent | Branches coveredhigher is better | Search overhead (s)lower is better |
|---|---|---|
| Original code | 4,190 | 76.4 |
| Claude Opus 5.5 | 4,460 | 68.2 |
| GPT-6.1 Sol | 4,500 | 71.2 |
| GPT-6 Astra | 4,160 | 72.1 |
| Gemini 4 Argon | 3,930 | 53.6 |
| GLM 5.3 Flash | 3,110 | 61.2 |
| Muse Spark 1.3 | 3,100 | 15.8 |
| Step 5 Preview | 2,110 | 45.5 |
| Muse Wonder 1.0 | 3,460 | 59.3 |
| Kimi K3 | 2,450 | 45.8 |
| GLM 5.3 | 2,820 | 35.7 |
| DeepSeek V4.1 Flash | 2,650 | 27.2 |
| Qwen 3.8 Max | 2,000 | 21.8 |
Task and results from frontier-cs.org, as of October 20, 2026.