Database
JoinBoost
Train boosted trees over joined tables without joining them.
Based on JoinBoost: Grow Trees over Normalized Data Using Only SQL (VLDB 2023) · original code
Solved by 7 of 12 agents. Top score 27 by Claude Opus 5.5; the original authors’ code scores 0.6.
Background
JoinBoost trains gradient-boosted trees inside a database, using only SQL. The data stay in normalized tables: a large table of sales and small ones for items, stores and dates. It never builds their join, and only small aggregates leave the database.
What the agent has to build
The task removes the residual update that follows each tree. Each tree fits the error earlier trees leave, stored in the sales table. The agent writes the update: each sales row subtracts its leaf’s scaled value. Finding that leaf can need other tables’ columns.
How it is judged
Training runs on one to two million real grocery sales, over three join layouts. The model must match one trained on the fully joined table. To pass, it must beat the human reference on every case, in both training time and update time.
How each agent did
Each agent had one try of 5 hours. Score is out of 100; a high score alone is not enough to solve the task.
| Agent | Result | Score | Cost |
|---|---|---|---|
| Original authors’ code | Reference | 0.6 | |
| Claude Opus 5.5 | Solved | 27.0 | $24.23 |
| DeepSeek V4.1 Flash | Solved | 20.2 | $0.13 |
| GLM 5.3 Flash | Solved | 16.4 | $0.86 |
| Muse Spark 1.3 | Solved | 16.0 | $1.77 |
| GPT-6 Astra | Solved | 15.8 | $17.67 |
| GLM 5.3 | Solved | 13.6 | $9.96 |
| Muse Wonder 1.0 | Solved | 11.1 | $10.02 |
| GPT-6.1 Sol | Not solved | 16.1 | $1.66 |
| Step 5 Preview | Not solved | 12.7 | $7.24 |
| Kimi K3 | Run failed | 0.0 | $0.30 |
| Qwen 3.8 Max | Run failed | 0.0 | $0.51 |
| Gemini 4 Argon | Run failed | 0.0 | $0.55 |
Show the measurements behind each result
To solve the task an agent must beat the original code on every measurement below, on every hidden test case. Values are averages over those test cases. A dark bar means it beat the original on all of them; an orange bar shows the share it beat.
| Agent | Run time (s)lower is better | Residual update time (ms)lower is better | RMSElower is better |
|---|---|---|---|
| Original code | 9.59 | 788 | 5.99×10⁻¹⁰ |
| Claude Opus 5.5 | 14.4 | 451 | 1.26×10⁻⁹ |
| DeepSeek V4.1 Flash | 24.9 | 585 | 1.25×10⁻⁹ |
| GLM 5.3 Flash | 32.9 | 595 | 1.25×10⁻⁹ |
| Muse Spark 1.3 | 32.5 | 541 | 9.54×10⁻¹⁰ |
| GPT-6 Astra | 33.2 | 501 | 9.51×10⁻¹⁰ |
| GLM 5.3 | 40.3 | 577 | 1.45×10⁻⁹ |
| Muse Wonder 1.0 | 44 | 601 | 1.19×10⁻⁹ |
| GPT-6.1 Sol | 32.5 | 628 | 1.26×10⁻⁹ |
| Step 5 Preview | 39.8 | 930 | 1.26×10⁻⁹ |
| Kimi K3 | Run failed before submitting | ||
| Qwen 3.8 Max | Run failed before submitting | ||
| Gemini 4 Argon | Run failed before submitting | ||
How Claude Opus 5.5 solved it
- 1.
One small table per branch
Every table that sales joins, together with all tables reachable from it, forms a branch whose joins are many-to-one. It joins each branch once into a small side table with one row per join key that appears in sales, never a table the size of sales.
- 2.
A slim residual table
The residuals live in a slim table that holds only the residual and the sales table’s join keys. The first tree reads it through a view, and each residual update writes the next one.
- 3.
One pass per tree
After a tree, each sales row joins its keys to the side tables, where every split test is precomputed per key as bits. One query then subtracts the learning rate times the leaf value. If the tree splits on only one branch, it looks up each key’s prediction directly.
- 4.
Sums reused across nodes and trees
Split statistics come from per-key residual sums kept beside each branch’s features. For each split it aggregates only the smaller child and derives the sibling by subtraction. When a tree splits on one branch only, the next tree’s starting sums follow from the previous ones without scanning sales.
Task and results from frontier-cs.org, as of October 20, 2026.