The comparison
nobody ran until now.
First surfaced in Tandemly Briefing — 2026-08-01.
Every published evaluation of multi-call orchestration compared a carefully tuned orchestration pipeline against a plain, unoptimized single-call baseline. Leins, Pelleriti, Gonnermann-Müller and Pokutta changed that. When every method, including the simplest single-call approaches, gets optimized with equal effort, the accuracy advantage of orchestration shrinks to 4.6 percentage points, at roughly two to four times the token cost.
A literature built on
unequal comparisons.
The case for multi-call orchestration has been built on comparisons that gave one side an advantage the other side never had: tuning.
Multi-call LLM orchestration covers a family of techniques that run the model more than once to improve a final answer. Self-Refine asks the model to critique and revise its own output. Best-of-N runs the model many times and selects the strongest answer. Debate pits multiple model instances against each other. All three have accumulated positive evidence in the published literature.
There is a problem with that literature. The orchestrated methods are usually prompt-tuned, temperature-tuned, and configuration-tuned before the comparison. The single-call baseline is often presented as-is, with the default system prompt and no optimization effort invested in it. The reported accuracy gap includes the gap from unequal tuning, not just the gap from the architectural difference.
This is not a minor methodological point. A well-tuned single-call prompt can behave quite differently from a default one. If a team decides to add an orchestration layer because the published literature shows it outperforming a single call, they may be paying for a gain they could have achieved with better prompting alone.
What happens to the orchestration advantage when every method, including the single-call baselines, is optimized to the same standard under the same budget? Is the gain from architecture, or is it partly from the asymmetry in how the two sides were prepared?
Five backbones, three domains,
equal optimization budgets.
The design is straightforward: take the same evaluation protocol, apply the same optimization effort to every method, and compare the results. The difficulty-stratification is the piece that makes the hardness question answerable.
The study compared three orchestration families against two single-call baselines. The orchestration methods were Self-Refine (the model critiques and revises its own answer), Best-of-N (the model answers multiple times and the best is selected), and Debate (multiple model instances argue toward a consensus). The baselines were task-only (a direct prompt with no chain-of-thought) and chain-of-thought (the model is asked to reason step-by-step before answering).
Every one of these five configurations was optimized using GEPA, a gradient-free procedure that adjusts prompts and hyperparameters, under an identical evaluation budget. Neither side was given more optimization effort than the other. This is the key departure from standard practice.
The evaluation covered five backbone models and three task domains: competitive programming problems, chess puzzles, and mathematics. The test items were stratified by difficulty so the authors could ask specifically whether harder problems show a larger orchestration benefit, a hypothesis that often serves as justification for the added cost.
Breaking the test set into difficulty bands allows a direct test of the hardness hypothesis: that orchestration earns its cost on the hard tail where single-call reasoning struggles most. If the accuracy gap from orchestration grows with difficulty, that hypothesis holds. If it does not, the common justification for adding orchestration on hard tasks is unsupported.
Small gains. Real costs.
No hardness payoff.
When the comparison is fair, orchestration still wins. But the margin is small, the token cost is substantial, and the two assumptions most commonly used to justify orchestration both fail.
Mixed-effects analysis found strong method-by-backbone interaction in all three benchmark domains. An orchestration strategy that outperforms the baseline on one model gives no reliable prediction about whether it will outperform it on a different model. A Self-Refine setup that works well on one backbone may underperform a single call on another. Validating an orchestration choice on one model is not sufficient before committing to it across a system with multiple backbone options.
Orchestration gains are positive in these results. The claim is not that multi-call methods are useless. It is that the cost case for orchestration is much weaker than the literature implies once both sides have been optimized equally. A gain of 4.6 pp at 2 to 4 times the tokens is a real engineering tradeoff, not a clear win. The study covers three orchestration families and three domains. Results may differ in other settings.
What this means
for building with LLMs.
The practical implication is not "abandon orchestration." It is "stop comparing a tuned orchestration stack to an untuned baseline and calling that an evaluation."
Where to go
from here.
Concrete steps for applying this work.