Agent Evaluation · Long-Horizon Tasks

Pass rate tells you
nothing about learning.

First surfaced in Tandemly Briefing — August 12, 2026.

Researchers ran seven frontier models through 36 long-horizon AI research and development tasks and recorded what actually happened inside each run: how the agent framed the problem, how it executed, and how it handled feedback. Final scores looked reasonable. The within-run picture told a different story.

Core finding
Agents on long-horizon R&D tasks behave like engineering optimizers: they assemble workable solutions from known techniques, but run-to-run variance is high and genuine methodological novelty is rare. Passing is not the same as learning.
scroll to explore
01The problem

Final scores hide
two different questions.

Most agent benchmarks answer one question: did the agent complete the task? The paper argues that for long-horizon work, you also need to ask whether the agent's process was sound, and whether it actually got better from experience.

Pass rate is a reasonable metric for narrow, well-defined tasks. Ask the agent to write a function, check whether the tests pass. Binary. Clear. But long-horizon research and development tasks are not like that. They involve multiple steps, intermediate decisions, and outputs where "completed the task" can mean anything from "stumbled into a workable answer" to "systematically solved it."

The field has largely been measuring the destination while ignoring the journey. An agent that passes eight out of ten runs with high variance is a very different system from one that passes eight out of ten with low variance. Both look the same on a leaderboard. One of them is reliable. The other is guessing well.

The second gap is experience reuse. If an agent completes a research task, does it perform better on the next one? Intuitively it should: the task is still in context, patterns should generalize. But nobody had run a controlled test to find out, and intuition about self-improvement is historically unreliable with language models.

What this paper sets out to measure

Not just whether agents pass long-horizon R&D tasks, but how consistently they pass them, what process they follow while doing it, and whether accumulated experience across tasks or within a task improves their decisions later.

02The experiment

Seven models, 36 tasks,
three behavioral axes.

The study ran seven frontier models through 36 long-horizon AI R&D tasks and captured within-run behavior with rule-based metrics, not just terminal pass/fail. Then ran controlled conditions to test whether experience actually transferred.

The setup is straightforward in principle and expensive in execution. Take seven frontier language models. Give each of them the same 36 long-horizon research and development tasks, the kind that require multi-step reasoning, iterative experimentation, and synthesis across multiple sources of evidence. Run each model multiple times to capture variance. But here is the key move: do not just record whether each run succeeded.

The researchers instrumented each run with rule-based behavioral metrics across three axes. Rule-based means no model is judging the model: these are structural checks on what kind of activity is happening at each stage of the run. The three axes each capture a different layer of the agent's behavior.

01
Solution Framing
How the agent characterizes the problem and forms an initial approach. Does it identify the relevant constraints? Does it articulate a method before executing?
02
Execution
How the agent carries out its plan. Does it follow through on the framing, or drift? Does it use the right tools and intermediate checks?
03
Feedback Control
How the agent responds to intermediate signals. When something fails or produces an unexpected result, does it adapt, or ignore it and continue?

On top of the within-run instrumentation, the researchers designed controlled comparisons to test experience reuse. Within a single task: does the agent's behavior improve across attempts at the same task? Across tasks: does completing one task make the agent more effective on the next? These are distinct claims that practitioners often conflate, and the study treats them separately.

Why rule-based metrics matter here

Using rule-based structural checks rather than a model-graded rubric removes one layer of stochastic noise from the measurement. The behavioral axes are observable from the run structure itself. This makes the findings replicable and cheap to extend to other evaluations.

03Findings

Optimizers, not researchers.

The behavioral picture was consistent across all seven models: solid execution mechanics, high run-to-run variance, solutions built from recombined known techniques, and very little evidence that agents got better at a task by having done it.

What pass rates suggest
Agents complete long-horizon research tasks reliably. Pass rates look reasonable. The range across models suggests capability differences. Standard benchmark story.
What process metrics reveal
Completion is noisy and solutions are mostly recombined. The same model on the same task produces substantially different results across runs. The strongest outputs assemble known techniques competently. Genuine methodological novelty is rare.
Finding 1: Run-to-run variance is substantial

When the same frontier model is run on the same long-horizon R&D task multiple times, the outcomes vary in ways that pass-rate averages smooth over. An agent with a 70% pass rate might be reliably solving seven out of ten tasks, or it might be inconsistently solving some of them with wide swings. These are architecturally different systems. The process metrics reveal which is which in ways that terminal scores do not.

Finding 2: Solutions recombine; they rarely invent

When agents produced strong solutions, those solutions were built from existing techniques assembled competently. The framing axis captured this directly: agents described their approach in terms of established methods and applied them to the task. When a task required genuine methodological novelty, agents mostly could not deliver it. They produced competent-looking outputs that, on close inspection, were recombinations of patterns from training. This is not a failure of execution. It is a limit of the framing layer: the agent knows how to apply what it has seen, but not how to extend beyond it.

Finding 3: Experience transfer is near-zero

The controlled comparisons were among the paper's sharpest contributions. Neither within-task experience (multiple attempts at the same task) nor cross-task experience (completing one task before starting another) produced measurable improvement in agent behavior on the process metrics. Agents reset. Each run starts from roughly the same behavioral posture as the last one. "The agent improved over time" is a claim that requires this specific kind of controlled test to support, and the data did not support it here.

Scope and limitations

36 tasks across seven models. Long-horizon AI R&D is a specific task family, and findings may not generalize to narrower or more structured task types where execution mechanics dominate and framing matters less. The study measures behavioral patterns, not the underlying mechanisms. The authors frame this as a diagnostic, not a verdict on what agents can ultimately do.

04Practical takeaways

Two questions your eval
is probably not asking.

Pass rates are necessary but not sufficient for long-horizon evaluation. The study implies two additional questions every agent eval should answer: how consistent is that pass rate, and does the agent actually improve from experience?

1
Report variance alongside mean pass rate
Run each eval task at three or more seeds and report the standard deviation next to the mean. An agent with 70% mean pass rate and low variance is a different product from one with the same mean and high variance. Both look identical on a single-run leaderboard. The variance tells you whether you have a system or a lucky draw.
2
Add a controlled experience-reuse condition
If you are claiming your agent "learns from prior runs" or "improves with use," you need an explicit test. Set up a cold-start condition and a prior-context condition on the same task family, and measure whether behavior differs on the process axes, not just final scores. The paper found near-zero transfer; replicate this finding on your own workload before trusting improvement claims.
3
Use rule-based process metrics, not just model-graded rubrics
The three-axis framework (Solution Framing, Execution, Feedback Control) captures behavioral patterns without requiring a model to judge a model. Rule-based structural checks are cheaper, more reproducible, and do not compound evaluation noise with generation noise. Start with framing: does the agent articulate a method before executing? That alone surfaces a meaningful signal.
4
Design for recombination, test for novelty separately
The engineering-optimizer finding is useful for task design. If your task requires assembling known techniques, current frontier models can handle it reasonably well, with variance. If your task requires genuine methodological novelty, treat that as a different capability class and evaluate it separately. Lumping them into a single pass-rate eval hides the gap between them.
5
Apply this framing to capability claims in vendor documentation
When a provider publishes a benchmark showing their model solves 70% of research tasks, ask: is that a mean over multiple seeds or a single run? Did they test experience transfer or assume it? Did they instrument the run or just record the output? These are cheap questions that significantly change how much a pass-rate number tells you.
05Further exploration

Where to go
from here.

Three concrete steps to apply the paper's measurement approach to your own agent work.

1
Read the paper
Li, Y., Yang, W., Tan, H., Huang, X., et al. (2026). Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development. arXiv:2608.13417. Full methodology and data in the appendix are worth reading if you plan to adopt the three-axis framework.
2
Pick one benchmark you already run and add multi-seed variance
You do not need to rebuild your eval from scratch. Take any existing long-horizon task and run it at five seeds instead of one. Compare the distribution against your prior single-run number. If variance is high, you now have a more honest number to report and a signal to investigate.
3
Build a minimal experience-reuse test
Pair every eval task with a "warm start" variant: give the agent access to a prior successful run trace and compare behavior with the same task from cold start. Measure whether solution framing, execution choices, or feedback responses differ. Even a three-task pilot surfaces whether transfer is happening at all.
4
Look at the related measurement-skepticism work
This paper sits in a cluster questioning standard eval assumptions. The distractor-aware truncation paper questions long-context degradation curves; the unified agent eval framework shows harness choice reorders leaderboards. Together they suggest that what most evals measure and what practitioners need to know are often different things.
5
Adopt the engineering-optimizer frame before specifying agent tasks
Before writing the task spec for any agent deployment, ask whether success requires assembling known techniques, or requires generating a method not well represented in the training distribution. Current agents do the first well and the second poorly. Knowing which kind of task you have shapes what success looks like and what failure modes to plan for.