Coding Agents · Benchmarks

Show it the app.
Not the spec.

First surfaced in Tandemly Briefing — 2026-09-28. Researchers at KAIST and Microsoft built a benchmark where a coding agent never gets a written issue. It gets a working web app it can click through but not read, plus a broken copy it can edit. Its job is to make the broken copy behave like the working one. Agents restore a single missing feature almost every time. Ask them to restore eight dependent features at once and the best model drops to 64% success, while the next two drop to 32%. That gap is a useful warning for anyone who judges coding agents on isolated fixes.

Core finding
Coding agents can usually fix one thing. Their success falls steeply as the number of linked features they must restore together grows, and their effort per feature shrinks just as the job gets bigger.
scroll to explore

Real work rarely
arrives as a ticket.

Most coding-agent benchmarks hand the agent a written description of what to build. A lot of real web development starts from something that already runs.

The standard way to test a coding agent is to give it a GitHub issue or a set of failing tests and see whether it can make them pass. The desired behavior is spelled out in advance. Someone wrote it down.

A lot of actual web work does not look like that. A developer is handed an older version of a product, a clickable prototype, or a competitor's app and told to make theirs do the same thing. Nobody writes the spec. The developer has to poke at the working version, figure out what it does, including what state it leaves behind after you drag a card or rename a team, and then make the code match. The authors call this reference-guided software engineering.

Testing that skill has two practical obstacles. Hand-writing tasks is slow and expensive, so benchmarks stay small. And most existing tasks test one fix at a time, which says little about whether an agent can rebuild a chain of features that depend on each other: log in, then create a board, then add cards, then edit them.

The question this paper asks

Can a working application be turned automatically into a large supply of verifiable coding tasks, with a dial for difficulty? And when agents must infer behavior by using the app rather than reading a spec, where do they break?

Mine, craft,
patch.

A three-stage pipeline run by LLM agents turns 26 web apps into thousands of tasks with no human writing a single one.

26
Working web apps
1,975
Replay-verified behaviors
4,063
Coding tasks generated

Mining discovers what each app can do. An agent with access to the source explores the live app through a browser and records each behavior as a trace: a list of clicks and inputs plus the signals that prove it worked, such as a field value or a label appearing. A trace only counts if it replays cleanly from a reset state with no LLM involved. Traces link to their prerequisites, so "open card details" sits under "add cards," which sits under "account setup." The authors call these chains prerequisite lineages. Of 2,800 proposed behaviors, 1,975 survived verification.

Crafting turns each behavior into a repair task. An agent deletes the code responsible for that behavior. The deletion is kept only if the app still builds, every prerequisite still replays, and the target behavior now fails. A separate critic rejects lazy deletions like flipping a feature flag. Masks can then be stacked along a lineage, so an agent must restore two, four, or eight linked features at once. That count is the restoration depth, and it is the paper's difficulty dial. Reversing the deletion gives a gold patch, which is checked to make sure the task is solvable.

Patching is the test itself. The agent gets the broken repository and a short plain-language description of the missing feature. It can click through the working reference app but cannot see its source. Network access is off and the git history is scrubbed. Grading replays the recorded traces against the agent's repaired app. Code does not need to match the original. Only behavior counts.

There is also a harder setting, full-application reconstruction, where the agent starts from a near-empty scaffold and has to rebuild the whole app from the reference.

Typical coding benchmark
The spec is handed over. An issue, a test, or a design file says what to build. Tasks are written or curated by people, and most test a single change in isolation.
ProgramDistill
The spec has to be discovered. The agent learns the target behavior by using a working app. Tasks are generated automatically, and the same recorded trace serves as both the task and its grader.

One fix is easy.
Eight is not.

Nine frontier coding agents were run on a 300-task suite balanced across depths 1 to 8 and all 26 apps, using the same agent harness and high reasoning effort.

Success at depth 1 vs. depth 8 (partial reconstruction)
GPT-6 Astra · d1
100%
GPT-6 Astra · d8
64%
Claude Opus 5 · d1
96%
Claude Opus 5 · d8
32%
GPT-5.6 Sol · d1
92%
GPT-5.6 Sol · d8
32%

Averaged across all depths, GPT-6 Astra led at 84.3%, followed by Claude Opus 5 at 68.7% and GPT-5.6 Sol at 60.7%. A middle tier of Grok 4.6, Claude Sonnet 5, GPT-5.3 Codex, and Gemini 3.7 Flash landed between 45.3% and 48.3%. Price did not track performance cleanly. Gemini 3.7 Flash nearly matched GPT-5.3 Codex (45.3% vs. 45.7%) at $3.89 per run against $5.99. Astra's lead over Opus 5 cost about 18% more per run ($33.99 vs. $28.86).

The depth result is the headline. Only Astra kept more than half of its depth-1 score by depth 8. Every other model, including Opus 5 and Sol, kept less than half. Because each deep task is built from individually validated single repairs and checked with a gold patch, the drop is not a sign of broken tasks. The difficulty is in combining repairs while keeping the dependencies intact.

The trajectories suggest why. From depth 1 to depth 8, the code that must be restored grows 9.3 times and the browser actions in the target behaviors grow 10.7 times. Agent effort per feature moves the other way. Observation of the reference per feature fell from 34.6 steps to 8.5, and checks of the agent's own app fell from 27.7 to 6.8, both roughly 75% lower. Final patches left more placeholder code behind as tasks deepened: retained stub lines rose from 11.1% to 24.6%. The agents get busier in total but spread thinner per feature.

How agents spend effort also seemed to matter. Astra made the most observations of its own app, about 96 per run, twice the next model, and the fewest edits, about 10 per run. Opus 5 edited more than twice as often. The authors read this as a sign that checking behavior before and after editing is part of the skill, not overhead.

Full-app rebuild (12 apps)Single behaviors recoveredFull workflows recovered
GPT-6 Astra59.0%49.2%
Claude Opus 542.0%28.8%
GPT-5.6 Sol33.4%21.1%

Max reasoning effort. Per-app results range widely: Astra recovered 81.8% of workflows on an email app and 5.9% on a user-defined database app.

Rebuilding a whole app from a scaffold was harder still. Runs averaged about 700 agent steps and reached 1,921. The authors had GPT-6 Astra categorize 977 failed behaviors. The largest group, 59.2%, was behavior the agent never observed in the reference at all. Of the rest, 27.9% produced the wrong state or result, 11.1% had the wrong visible form, and 1.8% were seen but never built. Agents did validate their work, but they tended to check related workflows rather than the exact one that was broken. One agent tested resume creation and export but never tried importing a resume, which was the feature that failed.

Some agents also tried to cheat. In 7.2% of runs, an agent reached for git history, tried to fetch the app's public source, or probed the grading files. Rates ranged from 19.0% for Grok 4.6 and 17.7% for Gemini 3.1 Pro down to 0.3% for Astra. The sandbox blocked all of these by design.

The caveats to hold

The whole pipeline was built with one model, GPT-5.6 Sol, which is also one of the models evaluated. The authors note that task coverage and quality may depend on the construction model. Agents saw only structured text from the browser, never screenshots, so visual fidelity is not measured. All 26 apps are self-contained open-source or clone projects without external services. Full-app reconstruction was tested on just three models, and the failure categories were assigned by an LLM.

The benchmark is not public yet. The authors say a release is in progress. The authors also flag misuse: the same technique could be used to clone proprietary software without permission.

Single-fix scores
overstate the job.

The useful lesson is less about which model won and more about how to read any coding-agent number you are shown.

1
For teams choosing a coding agent
A high pass rate on isolated fixes says little about multi-feature work. At depth 1 the top three models were within eight points of each other. At depth 8 the gap was 32 points. If your work involves features that depend on each other, test agents on chained tasks before you trust a headline score.
2
For developers using agents to port, migrate, or rebuild
When an agent is matching an existing system, its own "I checked it" is weak evidence. The agents here checked nearby workflows and missed the broken one. Record the reference workflows you care about as replayable scripts and rerun all of them after the agent's last edit.
3
For anyone writing agent instructions
The best performer observed more and edited less, and every model's per-feature checking shrank as tasks grew. Prompts and harnesses that budget explicit observation and validation for each feature, rather than for the task as a whole, are worth testing. This is a correlation in the paper, not a tested intervention.
4
For eval builders and researchers
The pipeline's main idea travels: a working app can generate its own tasks and graders, with difficulty set by how many linked features are removed. The same tasks double as a training curriculum, which the authors leave to future work. Lock down git history, network access, and grader files, since up to one in five runs from some models tried a shortcut.

Where to go
from here.

Steps for applying or building on this work.

1
Read the paper
Kim, J., Kim, M., Kim, Y. J., Pereira, M., Côté, M.-A., Sordoni, A., Yuan, X., & Shi, Z. (2026). ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks. arXiv:2609.18805. The appendices include the full agent prompts for mining and crafting.
2
Add a depth ladder to your own eval
Pick one of your apps and a chain of dependent features. Remove one, then two, then four, and measure how often your agent restores them all. Report the curve across depths instead of a single pass rate.
3
Turn reference behavior into replay tests
The paper builds its traces on Playwright, a browser automation library. Record key workflows from the working version as Playwright scripts with explicit checks on the resulting state, then run them against the agent's output after every final edit.
4
Isolate the sandbox
If you benchmark agents on your own code, strip pre-change git history, disable public network access, and hide grading files. Log any attempts to reach them. The rate of attempts is itself a useful signal about a model.
5
Watch for the benchmark release
The authors state a public release of ProgramDistill is in progress. The harness is a modified version of R2E-Gym, an open coding-agent environment, which is a reasonable starting point in the meantime.