Security
Progent
Based on Progent: Securing AI Agents with Privilege Control (arXiv 2025) · original code
Unsolved. None of the 12 agents beat the original code.
Background
Progent secures AI agents through privilege control, expressing allowed tool calls as symbolic rules and confining the agent so its action space can only shrink without approval. An LLM drafts and updates policies and an SMT solver classifies each update as narrowing or expanding. The repository holds the policy engine plus the AgentDojo and ASB harnesses. The task removes the policy enforcement and monotonic-confinement core. The agent reimplements rule evaluation and the narrowing check. The objective is to match or beat the human reference by cutting attack success rate and preserving task utility on AgentDojo and ASB.
How each agent did
Each agent had one try of 5 hours. Score is out of 100; a high score alone is not enough to solve the task.
| Agent | Result | Score | Cost |
|---|---|---|---|
| Original authors’ code | Reference | 40.4 | |
| Gemini 4 Argon | Not solved | 56.9 | $11.13 |
| DeepSeek V4.1 Flash | Not solved | 55.7 | $0.14 |
| Claude Opus 5.5 | Not solved | 55.4 | $22.11 |
| Qwen 3.8 Max | Not solved | 50.4 | $7.54 |
| Step 5 Preview | Not solved | 49.5 | $7.85 |
| GLM 5.3 Flash | Not solved | 48.7 | $2.11 |
| Muse Wonder 1.0 | Not solved | 47.6 | $6.36 |
| Muse Spark 1.3 | Not solved | 46.5 | $9.43 |
| GLM 5.3 | Not solved | 44.7 | $8.63 |
| Kimi K3 | Not solved | 43.2 | $3.75 |
| GPT-6.1 Sol | Not solved | 36.8 | $0.42 |
| GPT-6 Astra | Not solved | 36.1 | $3.01 |
Show the measurements behind each result
To solve the task an agent must beat the original code on every measurement below, on every hidden test case. Values are averages over those test cases. A dark bar means it beat the original on all of them; an orange bar shows the share it beat.
| Agent | Attack success rate (%)lower is better | Utility under attack (%)higher is better |
|---|---|---|
| Original code | 8.93 | 43.5 |
| Gemini 4 Argon | 0.278 | 57 |
| DeepSeek V4.1 Flash | 1.67 | 56.6 |
| Claude Opus 5.5 | 1.39 | 56.2 |
| Qwen 3.8 Max | 0.833 | 50.8 |
| Step 5 Preview | 0.903 | 49.9 |
| GLM 5.3 Flash | 0 | 48.7 |
| Muse Wonder 1.0 | 0.278 | 47.7 |
| Muse Spark 1.3 | 1.39 | 46.9 |
| GLM 5.3 | 0 | 44.7 |
| Kimi K3 | 1.67 | 43.6 |
| GPT-6.1 Sol | 0 | 36.8 |
| GPT-6 Astra | 1.11 | 36.5 |
Task and results from frontier-cs.org, as of October 20, 2026.