Machine Learning
Representation Fréchet Loss
Keep population statistics differentiable for a Fréchet-distance loss.
Based on Representation Fréchet Loss for Visual Generation (arXiv 2026) · original code
Unsolved. None of the 12 agents beat the original code.
Background
Fréchet distance compares two image sets by the mean and covariance of their features. This paper turns it into a training loss for image generators. Its statistics must cover many generated images, yet pass gradients to the current batch.
What the agent has to build
This task removes the feature queue that holds these statistics, and the agent writes it. It keeps past features or running estimates of their mean and covariance. It mixes in each new batch so gradients reach it, without keeping old computation graphs.
How it is judged
Each workload post-trains the authors’ released generator on a short schedule. Fresh images are then scored in six feature spaces, including Inception, DINOv2 and CLIP. To pass, the normalized Fréchet distance must be lower than with the human reference in all six spaces, on every workload.
How each agent did
Each agent had one try of 5 hours. Score is out of 100; a high score alone is not enough to solve the task.
| Agent | Result | Score | Cost |
|---|---|---|---|
| Original authors’ code | Reference | 3.7 | |
| Claude Opus 5.5 | Not solved | 7.2 | $11.94 |
| Qwen 3.8 Max | Not solved | 5.7 | $5.95 |
| Muse Wonder 1.0 | Not solved | 5.0 | $9.55 |
| DeepSeek V4.1 Flash | Not solved | 4.2 | $0.03 |
| Gemini 4 Argon | Not solved | 4.2 | $0.94 |
| GLM 5.3 | Not solved | 4.0 | $3.52 |
| Step 5 Preview | Not solved | 3.9 | $3.50 |
| GLM 5.3 Flash | Not solved | 3.8 | $0.42 |
| GPT-6 Astra | Not solved | 3.6 | $12.54 |
| Kimi K3 | Not solved | 3.6 | $1.27 |
| GPT-6.1 Sol | Not solved | 3.1 | $2.48 |
| Muse Spark 1.3 | Not solved | 2.9 | $3.98 |
Show the measurements behind each result
To solve the task an agent must beat the original code on every measurement below, on every hidden test case. Values are averages over those test cases. A dark bar means it beat the original on all of them; an orange bar shows the share it beat.
| Agent | Fréchet distance, CLIPlower is better | Fréchet distance, ConvNeXtlower is better | Fréchet distance, DINOv2lower is better | Fréchet distance, Inceptionlower is better | Fréchet distance, MAElower is better | Fréchet distance, SigLIPlower is better |
|---|---|---|---|---|---|---|
| Original code | 26.6 | 5.05 | 14.7 | 7.71 | 59.1 | 40.4 |
| Claude Opus 5.5 | 20.2 | 3.79 | 12.4 | 6.45 | 9.88 | 25.1 |
| Qwen 3.8 Max | 23.8 | 4 | 13.8 | 6.26 | 20.3 | 31.6 |
| Muse Wonder 1.0 | 23.9 | 4.25 | 13.9 | 6.69 | 31.4 | 33.4 |
| DeepSeek V4.1 Flash | 25 | 4.86 | 14 | 7.38 | 50.9 | 36.1 |
| Gemini 4 Argon | 25.3 | 4.73 | 14.4 | 7.16 | 47.6 | 37.7 |
| GLM 5.3 | 26.3 | 4.99 | 14.7 | 7.46 | 53 | 39.2 |
| Step 5 Preview | 26.8 | 5 | 14.4 | 7.5 | 53.2 | 40.1 |
| GLM 5.3 Flash | 26.1 | 5.18 | 14.4 | 7.73 | 57 | 41.2 |
| GPT-6 Astra | 27.3 | 5.28 | 14.8 | 8.14 | 61.3 | 41.6 |
| Kimi K3 | 26.2 | 5.33 | 14.8 | 7.89 | 64.9 | 43.9 |
| GPT-6.1 Sol | 29.6 | 5.9 | 15.8 | 8.58 | 79.5 | 49.4 |
| Muse Spark 1.3 | 30.4 | 6.36 | 16.1 | 9.15 | 88.2 | 52.5 |
How Claude Opus 5.5 approached it
- 1.
Three ways to keep the population
The queue supports the three estimators the training setups use: a ring buffer of the latest generated features, the same ring with running sums of the features and their outer products, and moving averages of the first and second moments, started from a warm start of generated images.
- 2.
Gradients through the batch only
To form the statistics, it combines the stored state, read without gradients, with the current batch, which keeps its graph. With running sums, it subtracts the rows about to be evicted and adds the batch’s sums; the mean and covariance follow from these moments.
- 3.
Updating without old graphs
After each step, it writes the detached batch over the oldest rows and adjusts the running sums by the difference, or moves the averages toward the batch. No computation graph outlives the step, and the state sits in buffers that save and load with the model.
- 4.
Double precision
All statistics are formed in 64-bit floats. Its own comments give the reason: in single precision, the covariance of a 50,000-row population loses its small eigenvalues, and rounding noise swamps the gradient that reaches the batch.
- 5.
A tracking window, its own idea
It argues that a long window, or a slowly decaying average, lags the generator, which then overshoots. So a long window gives way to an exponential window started from the warm start, holding about 1.5 times as many features as the feature dimension; for a ring buffer, repeated rows stand in for these weights.
Task and results from frontier-cs.org, as of October 20, 2026.