Security
LRR
Based on Repairing Regex Vulnerabilities via Localization-Guided Instructions (EACL 2026) · original code
Unsolved. None of the 12 agents beat the original code.
Background
Some regular expressions take exponential time on certain inputs, because a backtracking matcher tries every way to split the string. Whoever controls the input can then stall a server, an attack called ReDoS. LRR repairs such a regex with a language model. A symbolic tool, RegexScalpel, first finds the part of the regex at fault, and the instructions tell the model to rewrite that part.
What the agent has to build
The task removes the step that turns the part found into instructions. The agent writes it. The model, Qwen3-8B, and its decoding stay fixed, so only the instructions change.
How it is judged
The regexes are vulnerable ones from the Polyglot corpus: RegexScalpel repairs half of them on its own and fails on the other half. A repair counts when it compiles, ReDoSHunter finds no attack on it, and it still matches some of the original’s strings. To pass, a submission must beat the human reference on every group of regexes on eight counts: the shares well formed, repaired and no longer vulnerable; how much the regex grows and how much of it is edited; and three overlaps with the language of the original.
How each agent did
Each agent had one try of 5 hours. Score is out of 100; a high score alone is not enough to solve the task.
| Agent | Result | Score | Cost |
|---|---|---|---|
| Original authors’ code | Reference | 9.7 | |
| GPT-6.1 Sol | Not solved | 32.5 | $2.55 |
| Claude Opus 5.5 | Not solved | 29.0 | $11.92 |
| Muse Spark 1.3 | Not solved | 22.8 | $1.70 |
| Step 5 Preview | Not solved | 21.3 | $2.75 |
| DeepSeek V4.1 Flash | Not solved | 14.6 | $0.08 |
| Gemini 4 Argon | Not solved | 14.2 | $4.70 |
| GLM 5.3 | Not solved | 13.9 | $3.61 |
| Qwen 3.8 Max | Not solved | 13.0 | $9.46 |
| Muse Wonder 1.0 | Not solved | 8.2 | $3.24 |
| GLM 5.3 Flash | Not solved | 3.5 | $0.71 |
| GPT-6 Astra | Not solved | 0.0 | $9.95 |
| Kimi K3 | Run failed | 0.0 | $1.40 |
Show the measurements behind each result
To solve the task an agent must beat the original code on every measurement below, on every hidden test case. Values are averages over those test cases. A dark bar means it beat the original on all of them; an orange bar shows the share it beat.
| Agent | Jaccard similarityhigher is better | Normalized Levenshtein similarityhigher is better | Precisionhigher is better | Recallhigher is better | Repair ratehigher is better | Relative length increaselower is better | Vulnerability elimination ratehigher is better | Well-formed ratehigher is better |
|---|---|---|---|---|---|---|---|---|
| Original code | — | — | — | — | — | — | — | — |
| GPT-6.1 Sol | 0.755 | 0.747 | 0.8 | 0.776 | 0.43 | 0.0564 | 0.48 | 0.96 |
| Claude Opus 5.5 | 0.775 | 0.83 | 0.795 | 0.827 | 0.38 | 0.0143 | 0.46 | 0.98 |
| Muse Spark 1.3 | 0.742 | 0.769 | 0.79 | 0.758 | 0.33 | 0.0221 | 0.38 | 0.92 |
| Step 5 Preview | 0.647 | 0.761 | 0.714 | 0.691 | 0.36 | −0.0909 | 0.43 | 0.99 |
| DeepSeek V4.1 Flash | 0.786 | 0.833 | 0.806 | 0.815 | 0.19 | 0.105 | 0.23 | 0.98 |
| Gemini 4 Argon | 0.767 | 0.754 | 0.801 | 0.822 | 0.22 | 0.0637 | 0.24 | 0.98 |
| GLM 5.3 | 0.805 | 0.862 | 0.859 | 0.829 | 0.22 | −0.0603 | 0.25 | 1 |
| Qwen 3.8 Max | 0.844 | 0.869 | 0.877 | 0.863 | 0.18 | 0.018 | 0.21 | 0.99 |
| Muse Wonder 1.0 | 0.779 | 0.824 | 0.843 | 0.804 | 0.18 | −0.0181 | 0.23 | 0.97 |
| GLM 5.3 Flash | 0.839 | 0.909 | 0.886 | 0.846 | 0.09 | −0.00765 | 0.11 | 0.99 |
| GPT-6 Astra | 0.239 | 0.813 | 0.278 | 0.243 | 0 | 0.158 | 0.16 | 0.94 |
| Kimi K3 | Run failed before submitting | |||||||
Task and results from frontier-cs.org, as of October 20, 2026.