The plan failed.
or did the execution?
Most agent benchmarks collapse planning and execution into a single pass/fail score. Researchers at Tongji, Shanghai AI Lab, and six other institutions built a benchmark that pulls them apart, then added a fifth dimension almost nobody measures: whether agents know when to stop.
First surfaced in Tandemly Briefing — 2026-06-04.
You can't fix what
you can't attribute.
When an agent fails a task, the failure could have started in the plan or in the execution of a correct plan. Most benchmarks give you no way to tell the difference.
Imagine an agent that consistently fails to complete multi-step tasks. The obvious question is: what's breaking? Is the agent producing bad plans before it touches a single tool? Or is it planning correctly but fumbling the execution: calling tools in the wrong order, misreading intermediate results, recovering poorly from unexpected errors?
These two failure modes call for completely different fixes. A bad planner needs better reasoning, clearer task decomposition, or more deliberate chain-of-thought prompting. A bad executor needs better error handling, schema validation, and retry logic. If your evaluation produces a single end-to-end pass/fail score, you have no way to tell which of these you are actually dealing with. You might spend months improving the execution layer while the planner is the problem, or vice versa.
There is a second gap that most benchmarks ignore entirely. Production agents do not receive only solvable tasks. They receive requests that assume capabilities they do not have, instructions that contradict each other, and goals that are simply out of scope. A well-calibrated agent should recognize these and surface them rather than spinning toward an impossible answer. A benchmark that includes only solvable tasks gives you no signal on whether your agent does this. An agent that always attempts and always fails on infeasible tasks looks identical to one that never receives them.
Agent Planning Benchmark (APB) is the first diagnostic that cleanly separates planning failure from execution failure, and one of the first to measure infeasibility detection at scale. It does not replace end-to-end evals. It sits alongside them to tell you where inside the agent the failure lives.
4,209 cases,
five diagnostic settings.
The benchmark is built around five settings, each designed to isolate a different dimension of planning capability. Twelve multimodal models were evaluated across all five.
The team built APB with 4,209 multimodal test cases spanning 22 domains. The breadth reduces the risk that domain familiarity explains performance differences: an agent that has seen many cooking-task trajectories should not get credit for that on a logistics planning task.
The five-setting structure is the core methodological contribution. Each setting isolates a specific planning capability and keeps everything else fixed, so failure patterns point directly at the layer being probed.
A single end-to-end pass/fail score hides performance profiles. An agent could score 60% overall while performing at 85% on holistic planning and 30% on infeasibility detection. Those two agents need very different interventions. The five-setting structure surfaces that separation rather than averaging it away.
Three consistent
weak points.
Across 12 models and 4,209 cases, three patterns held. None of them show up cleanly in end-to-end completion benchmarks.
On holistic planning tasks with more steps, performance fell in ways that shorter-task benchmarks would not surface. Models that looked capable on 3-step tasks showed meaningful degradation on 8-step tasks. The planning layer, not execution, was the source of the degradation: agents committed to plans early that could not be revised effectively when conditions changed.
This matters because most internal agent evals use representative tasks from the product's most common workflows, which are often short. Long-horizon planning failure is invisible until a customer tries something more complex.
When the available tool catalog was expanded with irrelevant options, planning accuracy fell across most models tested. This suggests that agents are using tool availability as a proxy for task feasibility: if a tool exists that could relate to the task, the agent anchors on it even when it should be excluded from the plan.
In production, tool catalogs grow over time. An agent that performed well with 10 tools at launch may degrade when given 30 as new integrations are added. The benchmark's third setting makes this degradation measurable before it surprises teams.
Models showed consistent over-confidence on tasks designed to be impossible. Rather than declining, they attempted. This failure does not appear on task-completion metrics because those metrics only include tasks that can be completed. A team relying on completion rate as their primary eval metric has no signal on this dimension at all.
Refusal calibration matters because the cost of a false-positive attempt is different from the cost of a false-positive completion. An agent that burns its full token budget on an infeasible task wastes compute and delays the user's recognition that the task cannot be done.
APB was published as a diagnostic framework, not a deployment ranking. Exact model scores are available in the paper. The benchmark's value is primarily structural: teams should adapt the five-setting diagnostic pattern to their own task distributions rather than treating APB's published numbers as a selection criteria for a different use case.
What this changes
for agent builders.
The benchmark results matter less than the diagnostic structure. The five-setting pattern is something any team can apply to their own task distribution with a small number of hand-written test cases.
Where to go
from here.
Concrete starting points for teams who want to apply this diagnostic approach.