First surfaced in Tandemly Briefing — 2026-08-20.

Agent Evaluation · Algorithmic Self-Improvement

Most agents never touch
how the model learns.

Researchers built a benchmark with one question: can an LLM agent redesign a training algorithm, not just tune parameters around one? Across 29 configurations of 6 systems, the best scored 0.250 on a scale where the existing shipping algorithm is 0.1. The gap is not capability. It is willingness to attempt the hard thing.

Core finding
Submissions that change how the model learns average 0.226. Those that only adjust parameters average 0.126. Raising reasoning budget moves the mechanism-attempting fraction from 8% to 64%.
scroll to explore
01The problem

Benchmarks measure
everything except this.

Agent benchmarks test code writing, tool use, planning, retrieval. Almost none ask whether an agent can change the training algorithm sitting underneath an AI pipeline. That gap matters for anyone building or evaluating self-improving systems.

Claims about AI "self-improvement" circulate frequently in research and in product marketing. The usual evidence comes from agents that collect better training data, search hyperparameter space more thoroughly, or refine prompts. These are real capabilities, but they all take the algorithm as given. They work within the existing objective function and update rule.

Redesigning the training algorithm is categorically different. Writing a new optimizer, restructuring the training loop, or proposing a different loss function requires understanding what the current algorithm does, diagnosing where it fails mechanically, and proposing a replacement that still fits the evaluation setup. No major benchmark measured whether agents could do this at all.

A second problem is commensurability. Different training algorithm families produce results on completely different scales. An accuracy metric for a reinforcement learning optimizer has nothing in common numerically with a precision score for a language-model fine-tuner. Any benchmark that mixes algorithm families needs a normalization strategy that makes cross-domain comparison honest without collapsing variation within each domain.

The question this paper asks

Can an LLM agent read a research codebase, understand what its training algorithm does, and submit a modified version that measurably improves on the original? Not by tuning parameters. By changing the mechanism.

02The experiment

Ten algorithms, frozen.
Four hours to rewrite.

AI4AI-Bench built 10 frozen research repositories, each centered on a different training algorithm family. The evaluators, datasets, and scoring procedures cannot be changed. The agent can only change the algorithm.

Each repository represents a different training algorithm domain: reinforcement learning, continual learning, neural architecture search, optimization, and several others. "Frozen" means the rest of the system is locked. The agent cannot adjust the evaluation criteria or substitute a different dataset.

The agent receives four hours of compute on a single B300 GPU to read the codebase and submit a modified training algorithm. After submission, the repository runs the agent's version from scratch, with up to 12 hours to complete. A fixed evaluator scores the result against the original shipping algorithm running under identical conditions. The evaluator is hidden from the agent during the rewrite window.

Because the 10 domains produce completely incommensurable metrics, every result maps to a normalized score anchored at three points: 0 is what an uninformative random model produces, 0.1 is what the repository's own shipping algorithm produces, and 1.0 is the theoretical task optimum. This makes a comparison across RL and architecture search and continual learning readable without requiring domain expertise to interpret any individual number.

The benchmark ran 29 configurations drawn from 6 agent systems, recording every submission along with a behavioral label: did the agent's output actually change how the model learns, or did it adjust parameters and reorganize code without altering the algorithmic mechanism? All task suites, evaluators, and scored submissions are publicly released.

Why the normalization matters

A score of 0.15 looks different depending on whether 1.0 is easy or impossibly hard to reach. By anchoring at the shipping algorithm (0.1), the benchmark gives every number a concrete reference: anything above 0.1 is better than the existing baseline, and the gap to 1.0 is a real measure of headroom. This design is reusable beyond AI4AI-Bench for any evaluation that mixes incommensurable outcome types.

03Findings

The ceiling is 0.250.
The typical score is 0.166.

Neither number is encouraging on its own. What makes the results useful is the behavioral split underneath them: submissions that try to change the mechanism and those that do not tell very different stories.

0.10
Shipping baseline
The score the repository's own algorithm produces, used as the reference anchor for all comparisons.
0.166
Mean across all configs
Across 29 configurations of 6 systems. Just above the shipping baseline, but not by much.
0.250
Best system
The strongest configuration closes about a fifth of the distance between the shipping algorithm and the theoretical optimum.

The headline averages understate what the data shows. The paper logged a behavioral classifier alongside each submission: did the agent's code actually change how the model learns (a new objective, a restructured update rule, a different algorithmic mechanism), or did it adjust parameters and reorganize surface-level code while leaving the algorithm intact?

The two groups score very differently. Submissions that change the mechanism average 0.226. Submissions that do not average 0.126. The capability gap between a strong and a weak agent system is real, but it is smaller than the gap produced by whether the agent attempts the mechanism change at all.

Parameter-only submissions
Average score: 0.126. Adjust configuration values, reorganize code, sometimes re-implement the same logic in slightly different form. Do not change what the algorithm optimizes or how it updates. This describes most submissions from most systems.
Mechanism-change submissions
Average score: 0.226. Propose a different objective, restructure the update rule, or change what the algorithm is fundamentally doing. These submissions are rare at low reasoning budgets: only 8% of submissions with standard reasoning and 64% with higher reasoning.
What reasoning budget actually buys

Raising reasoning effort pushes the mean score from 0.094 to 0.196 and the mechanism-attempting fraction from 8% to 64%. Spending more on reasoning does not primarily produce more accurate parameter guesses. It produces willingness to attempt the harder structural change. The two are not the same, and treating them as interchangeable is one reason low-reasoning-budget agents tend to underperform on open-ended tasks.

Scope and limitations

Ten training-algorithm families, 29 configurations, 6 systems. Frozen evaluators and fixed compute budgets mean the benchmark captures a specific slice of algorithm-redesign capability: what agents can do in four hours on one GPU against a codebase they cannot modify around their submission. Real algorithmic work involves ambiguous objectives and code not written to be read by an agent. The 0.250 ceiling here is a calibrated snapshot, not an absolute limit.

04Practical takeaways

What this means
for building agent systems.

The paper covers algorithmic capability, but three of its design choices apply beyond this specific benchmark to any team building, prompting, or evaluating autonomous agents.

1
For teams evaluating agent "self-improvement" claims
Check whether the agent changes the mechanism or the configuration. A classifier that labels each submission "mechanism change" vs. "parameter adjustment" is cheap to add and reveals behavioral patterns hidden by aggregate scores. The two groups perform nearly 2x apart on normalized outcomes.
2
For anyone prompting agents on open-ended optimization work
Raise the reasoning budget before rewriting the prompt. The jump from 8% to 64% mechanism-attempt rate under higher reasoning is unusually clean evidence. If an agent keeps returning safe configuration changes on an open-ended task, the first intervention to try is more reasoning budget, not a different framing.
3
For benchmark designers working across incommensurable domains
The 0/0.1/1.0 normalization anchor is worth borrowing. Anchoring at the existing shipping baseline (0.1) gives every number a concrete reference point and makes cross-domain leaderboards readable without domain expertise for each entry. Any evaluation that mixes outcome types with incompatible scales can apply this pattern.
4
For frontier AI teams working on recursive self-improvement
The current state is that the best configuration closes about a fifth of the distance between the shipping algorithm and the theoretical optimum. That is a real foothold, not a negligible number. But it also means four-fifths of the headroom is still untouched by current systems. Treating the task as solved because agents can outperform the shipping baseline would be premature.
05Further exploration

Where to go
from here.

If you want to go deeper or apply this work.

1
Read the paper and data
Chi, Y., Li, W., Hong, D., Wang, X. et al. (2026). AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement. arXiv:2608.20318. The task suite, evaluators, and all scored submissions are publicly released.
2
Run your own agents against the frozen repositories
The frozen evaluation setup means you can test any agent system without modifying the benchmark. A good starting point is the domain most relevant to your existing work, running at two reasoning-budget levels to measure the behavioral shift directly.
3
Add a mechanism-vs-parameter classifier to your agent submission logs
A simple classifier that labels whether each submission touches algorithmic mechanism or only configuration is inexpensive to build and surfaces behavioral patterns that aggregate task-success rates hide. Track the mechanism-attempt rate over time alongside the outcome score.
4
Test reasoning budget before rewriting prompts
Double the reasoning budget on any open-ended optimization task where the agent keeps returning conservative changes, and measure whether the mechanism-attempt rate climbs toward the 64% mark seen in the paper. If it does, budget is the binding constraint.
5
Borrow the normalization scheme for your own cross-domain evals
Apply the 0 (uninformative) / 0.1 (existing baseline) / 1.0 (task optimum) anchor to any multi-domain evaluation where raw metrics are incommensurable. Anchoring at the existing tool or method, not at zero, makes relative progress visible and prevents high-variance domains from dominating the aggregate.