Machine Learning
RAGEN
Turn multi-turn agent trajectories into training batches.
Based on RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning (arXiv 2025) · original code
Unsolved. None of the 12 agents beat the original code.
Background
RAGEN trains language models with reinforcement learning to act over many turns. The model sees a puzzle, replies with moves, then gets the next view and a reward. Training needs these games packed into token batches that mark what to learn from.
What the agent has to build
The task removes the context manager that does this, and the agent writes it. It builds each prompt from the real history and parses replies into moves. It packs whole games into padded rows, with masks and with each reward on the right token. Games from the same start stay in one group.
How it is judged
Each run plays Sokoban, a box-pushing puzzle, with a small model, then takes one training step. To pass, every group of games must beat the human reference on both puzzles solved and average reward.
How each agent did
Each agent had one try of 5 hours. Score is out of 100; a high score alone is not enough to solve the task.
| Agent | Result | Score | Cost |
|---|---|---|---|
| Original authors’ code | Reference | 11.7 | |
| GPT-6.1 Sol | Not solved | 18.5 | $0.98 |
| Claude Opus 5.5 | Not solved | 18.5 | $9.19 |
| Muse Spark 1.3 | Not solved | 11.7 | $2.76 |
| DeepSeek V4.1 Flash | Not solved | 11.7 | $0.04 |
| GLM 5.3 | Not solved | 11.7 | $4.87 |
| GLM 5.3 Flash | Not solved | 11.7 | $1.72 |
| Kimi K3 | Not solved | 11.5 | $1.96 |
| Muse Wonder 1.0 | Not solved | 11.5 | $3.34 |
| GPT-6 Astra | Not solved | 11.3 | $8.00 |
| Gemini 4 Argon | Not solved | 11.3 | $6.08 |
| Qwen 3.8 Max | Run failed | 0.0 | $0.08 |
| Step 5 Preview | Run failed | 0.0 | $2.63 |
Show the measurements behind each result
To solve the task an agent must beat the original code on every measurement below, on every hidden test case. Values are averages over those test cases. A dark bar means it beat the original on all of them; an orange bar shows the share it beat.
| Agent | Average episode rewardhigher is better | Success ratehigher is better |
|---|---|---|
| Original code | −0.325 | 0 |
| GPT-6.1 Sol | 0.383 | 0.0833 |
| Claude Opus 5.5 | 0.383 | 0.0833 |
| Muse Spark 1.3 | −0.325 | 0 |
| DeepSeek V4.1 Flash | −0.308 | 0 |
| GLM 5.3 | −0.325 | 0 |
| GLM 5.3 Flash | −0.308 | 0 |
| Kimi K3 | −0.408 | 0 |
| Muse Wonder 1.0 | −0.408 | 0 |
| GPT-6 Astra | −0.483 | 0 |
| Gemini 4 Argon | −0.475 | 0 |
| Qwen 3.8 Max | Run failed before submitting | |
| Step 5 Preview | Run failed before submitting | |
How Claude Opus 5.5 approached it
- 1.
Finding the turns
The packed row is one chat: the system prompt, then the user and the model in turn. It counts the chat template’s turn-start tokens along the row, so every token knows which turn it belongs to. The loss mask covers everything after the system prompt, and the response mask covers only the model’s replies.
- 2.
Placing the reward
A game’s reward goes on its last token or, with rewards per turn, on the last token of each reply. Masks and rewards then shift by one token, so that each lines up with the token the model predicts.
- 3.
Parsing replies
Each reply goes to the repository’s own parser, with the configured separator and the limit on moves per turn.
- 4.
Normalizing in groups
Each game’s total reward, with any format penalty, is normalized within its group of games from the same start: the group mean is subtracted and the result divided by the group’s spread. A group whose rewards are all equal gets 0, so it teaches nothing.
- 5.
A shorter history, its own idea
The small model copies whatever earlier answer it sees, broken ones included. So a prompt keeps the previous turn only when that turn produced valid moves, and otherwise shows the current view alone. In its own offline Sokoban runs, this raised well-formed answers on later turns from about 45% to 75%, and solved games from 5.5% to 9.6%.
Task and results from frontier-cs.org, as of October 20, 2026.