Evaluation
How you measure whether the system actually works: equal-compute baselines, instrumentation that catches silent over-budgeting, and benchmarks that survive scrutiny.
No practices in this stage match the current filters.
-
WhenYou’re evaluating a multi-agent stack against a single-agent baseline, or assessing a vendor’s claim that their multi-agent design improves quality.
UseA single-agent baseline at equal thinking-token budget before adopting multi-agent. Hold the total reasoning compute constant across both architectures. If single-agent matches the multi-agent system at equal compute, the multi-agent overhead is not buying you anything on this task.
EvidenceAcross three model families and five multi-agent architectures (sequential, debate, ensemble, parallel-roles, subtask-parallel), the multi-agent advantage on multi-hop reasoning largely vanished when total thinking-token budgets were normalized. Single-agent reasoning matched or beat every multi-agent variant at every meaningful budget level above 100 tokens.
-
WhenYou’re benchmarking reasoning systems and relying on a vendor’s reported reasoning-budget parameter to control compute.
UseDirect instrumentation of actual thinking-token usage instead of trusted API budget caps. Log the realized token counts for every system under test. Treat documented budget parameters as soft hints unless you have verified they behave as hard caps on the specific model and version you’re using.
EvidenceThe authors of the equal-budget study found that Gemini’s thinkingBudget parameter does not behave like a hard cap. Actual visible-thought output often fell well below the requested budget, and API-reported token counts did not always match the visible reasoning text. Multi-agent systems that make multiple calls under the same nominal budget can be silently over-credited, distorting any comparison built on the parameter alone.
-
WhenYou are evaluating a generative image model and reporting results, or selecting a generator for a downstream task such as data augmentation or synthetic training data creation.
UseA coverage metric (such as IRS) alongside FID. Measure how much of the training distribution the model’s outputs actually span, not just how realistic individual images look. A model can score well on FID while silently skipping 20% or more of the real distribution; coverage metrics detect failures that FID cannot.
EvidenceDombrowski et al. (CVPR 2025) measured state-of-the-art unconditional image generators against their training distributions using IRS, a retrieval-based coverage metric. No tested model exceeded 77% coverage. FID scores for these same models were competitive. Standard diversity-proxy metrics built on Inception v3, DINOv2, and CLIP failed to detect the gap that IRS surfaced.
-
WhenYou are evaluating AI tools intended to support learning, whether for employee onboarding, customer education, or formal instruction, and need to compare options.
UseInclude a pre-test and post-test in your evaluation alongside any usability or satisfaction measures. Engagement scores and NPS do not substitute for learning measurement. A tool that scores high on satisfaction while producing no learning gain is not doing the educational job. Even a short quiz before and after a study session gives directional signal on actual retention.
EvidenceThe Taneja et al. (AIED 2026) study measured learning outcomes through post-test scores alongside user experience surveys. The ranking of systems on outcomes matched their ranking on experience, but experience scores alone would not have quantified the learning benefit. Controlled studies in educational AI frequently find engagement and outcomes diverge; this study happened to find them aligned, which is not the default assumption to make.
-
WhenYour team uses one or more AI coding agents to open pull requests and you want to assess whether those contributions are adding durable value.
UseCode churn rate as a primary evaluation metric alongside merge rate. Tag commits or PRs by source, then check at 30- and 90-day intervals whether lines introduced by agent PRs are still present or have been rewritten. Treat high churn as a signal to investigate whether the agent is solving problems at the right level of abstraction, not a reason to stop using agents.
EvidencePopescu et al. (2026) tracked 110,000 open-source pull requests from five coding agents and found that agent-authored code churned at higher rates over time than human-authored code, across all five agents, even for agents whose merge rates exceeded the human baseline. Merge rate and code longevity diverged consistently across the dataset.
-
WhenYou are building or evaluating an agent that retrieves stored information (user preferences, organizational policies, external facts) and uses it to generate responses or take actions.
UseImplicit-conflict test cases as a dedicated eval category. For each major memory type your agent stores, write three to five scenarios where later context implies the stored fact is outdated without stating it directly. Check whether the agent continues acting on the old fact. Treat any failure as a memory validity gap, not a retrieval gap: the agent retrieved correctly and reasoned incorrectly about whether to trust what it found.
EvidenceChao, Bai et al. (2026) built STALE, a 1,200-query benchmark probing three axes of memory staleness detection. The best frontier model evaluated scored 55.2% overall. Failures clustered on implicit conflict cases, where invalidation required multi-step inference rather than text matching. Frontier models handle explicit contradictions reasonably and fail most often when the invalidation is distributed across context rather than stated directly.
-
WhenYou are building evaluation infrastructure for LLM agents that use stateful tools and need to reproduce specific failure scenarios across model versions or configuration changes.
UseSeed-controlled failure injection rather than random perturbations. Assign each failure scenario a fixed seed so the same API error, timeout, or unexpected response can be replayed identically across every model, configuration, and release you test. Build a regression suite around your known failure seeds. A failure that appeared in one release and was fixed should be a permanent test case, not a one-time observation.
EvidenceLi, Yang, Wang et al. (2026) demonstrated that seed-controlled environmental perturbations, including API failures, allow valid cross-model comparison in a stateful tool environment. Randomized failure injection produces noise that obscures whether differences between models reflect capability or luck. Seeded injection produces reproducible, debuggable results that generalize across conditions.
-
WhenYou are evaluating whether a reasoning optimization (a new decoding strategy, prompt format, or inference technique) genuinely improves efficiency, and your only current metric is token count or latency.
UseAccuracy measured alongside token reduction as a joint success criterion. A technique that cuts tokens but degrades accuracy is a cost-accuracy tradeoff to be characterized, not a win to deploy. A technique that cuts tokens and maintains or improves accuracy represents a genuine efficiency gain. Report both metrics in any evaluation, and establish which accuracy benchmarks are representative of your production task distribution before running the comparison.
EvidenceLaTER’s training-free variant reduced tokens 16 to 32% on Qwen3-14B while matching or improving accuracy on AIME 2025 (70.0% to 73.3%) and MATH-500 (87.4% to 87.6%). The fine-tuned variant achieved 80.0% on AIME 2025 at 33% fewer tokens than standard CoT fine-tuning. In all three conditions, token reduction and accuracy moved in the same direction, confirming the efficiency gain was real rather than a tradeoff artifact. Plotting the cost-accuracy frontier across candidate strategies and using it to set token-budget guardrails in the deployment configuration captures this operating point before committing compute to production.
-
WhenYou are evaluating a routing strategy between LLM inference and agent execution and want to know whether it will hold up as your query distribution shifts over time.
UseA three-split evaluation structure: in-domain queries (similar to your routing history), paraphrased queries (same task, different wording), and out-of-domain queries (different topic areas). Report performance separately on each split. A router that performs well only on in-domain queries will degrade silently in production as the query distribution drifts. If OOD performance is substantially lower than in-domain, invest in expanding the routing history to cover more of the expected distribution before deployment.
EvidenceRouteBench, the benchmark introduced by Wang, Qiu et al. (2026), showed that routers tested only on in-domain queries overestimate generalization. BoundaryRouter’s rubric-guided reasoning generalized better than retrieval-only approaches on paraphrased and OOD splits, but OOD performance still degraded relative to in-domain. The three-split structure revealed this gap; single-split evaluation would have hidden it.
-
WhenYou are planning or reviewing the safety evaluation strategy for a memory-equipped LLM agent before or during production deployment.
UseDefine a fixed probe set at deployment time and run it against the agent’s accumulated memory state at regular intervals (30, 60, and 90 days at minimum), with a NullMemory counterfactual baseline for each run. Any safety violation the memory-equipped agent produces that the baseline agent does not is a memory-induced violation. If the rate is non-zero and growing over time, you have confirmed longitudinal safety drift in your specific system and need to investigate memory architecture or memory hygiene before expanding deployment.
EvidenceAl-Tawaha, Gu, Niu, Jia and Jin (Virginia Tech, UC Berkeley, and UIUC, 2026) ran a trigger-probe protocol across 8 memory architectures and 3 deployment scenarios (document management, scheduling, and email correspondence). Memory-induced violation rates climbed monotonically with exposure length across all configurations tested. Order-randomization experiments confirmed the effect is driven by accumulated content, not task sequence. Point-in-time safety evaluations at deployment did not predict in-production safety behavior.
-
WhenYou are reviewing an interpretability finding, an internal audit report, or a vendor’s interpretability tool output, and need to decide whether to act on it or treat it as exploratory.
UseApply a two-question check before acting on any interpretability finding. First: what specific intervention does this finding enable? Second: has that intervention been tested and confirmed to produce the expected behavioral change without degrading other model capabilities? If either answer is vague or absent, treat the finding as exploratory research. Do not let it drive decisions about model behavior in production until both criteria are met.
EvidenceOrgad, Barez, Haklay et al. (2026) surveyed the interpretability literature and found that most work satisfies at most one of two criteria: concreteness (the finding specifies what to change) and validation (the proposed change has been tested and confirmed). The authors argue that the field has built most of its reward structure around concreteness without equivalent pressure on validation, creating a gap between how findings are evaluated in research and what would be required to trust them in production.
-
WhenAn existing inference-time committee (best-of-N, critic layer, self-consistency) is not delivering the accuracy gain you expected, and you need to diagnose which part of the design is the weak link.
UseThe four-property diagnostic in sequence on 20 to 30 representative tasks. First, check coverage: does any model in the pool ever produce a correct answer? If coverage is below 50%, add more diverse proposers or vary prompting strategies before changing anything else. Second, check identifiability: when a correct answer is in the pool, how often does the system select it? If identifiability is low, the local verifier or critic is the bottleneck, not proposal quality. Third, check progress: do individual proposals advance meaningfully toward a solution, or are they stalling at similar intermediate states? If progress is low, restructure the task decomposition. Fourth, check diversity: are pool outputs substantively different, or just stylistic variations of the same approach? Low diversity means adding more calls to the same model with the same prompt will not help. Address each bottleneck in order.
EvidenceThe boosting framework from Sunkaraneni et al. (2026) separates committee performance into four measurable quantities: proposal coverage, local identifiability, progress, and diversity. The paper shows these are independent failure modes with different root causes and different fixes. The gap between the orchestrated committee (76.4%) and the oracle ceiling (79.0%) on SWE-bench Verified is directly attributable to identifiability failures: cases where a correct answer was present but the critic-comparator could not reliably select it.
-
WhenYou are selecting or evaluating a search agent for a product where users often have incompletely specified intents, such as research tools, recommendation flows, or decision-support applications.
UseAdd a multi-turn clarification phase to your search agent eval before optimizing retrieval accuracy. Run tasks where the query is deliberately vague and measure two things separately: how many clarifying questions the agent asks before retrieving, and how completely the final output covers the target information. Optimize for clarification quality first. A good clarifying question unlocks information that a fast retrieval hop cannot reach regardless of retrieval method quality.
EvidenceVibeSearchBench (Xiaohongshu Inc. 2026) tested frontier search agents on 200 bilingual tasks where users progressively disclose intent through multi-turn dialogue rather than specifying it upfront. The best frontier model scored 30.3 Triplet F1, recovering roughly a third of the target information. Clarification behavior was the binding constraint: agents that asked better questions in early turns outperformed agents that retrieved faster but skipped clarification. Standard single-turn benchmarks on the same models show substantially higher scores, confirming that the clarification gap is the source of most of the real-world performance shortfall.
-
WhenYou are designing evaluation tracks for a search agent and all existing tracks use fixed answer schemas: document IDs, ranked lists, or structured JSON responses.
UseAdd at least one eval track with schema-free output, where the agent must discover what the correct output structure is through dialogue, not match a template. A knowledge graph, a free-form summary that the judge scores for coverage, or any output whose shape must be inferred from user intent all qualify. Use this track to measure content recovery, not format-matching. Run both fixed-schema and schema-free evals on the same models and report both scores; the gap between them tells you how much of your agent’s benchmark performance depends on being handed the answer format.
EvidenceVibeSearchBench evaluates agents using directed knowledge graphs as ground truth, with Triplet F1 as the metric. The graph has no preset shape: agents must discover the correct output structure by learning what the user actually needs. The 30.3 F1 ceiling across frontier models (Xiaohongshu Inc. 2026) demonstrates that format-matching ability, which inflates scores on fixed-schema benchmarks, does not transfer to schema-free tasks. The benchmark is open at github.com/VibeBench/VibeSearchBench.
-
WhenYou are planning to scale a parallel agent pool to improve accuracy and want to know whether the investment will pay off.
UseMeasure pairwise error correlations between agents on a held-out test set before scaling. If correlations are consistently high (above ~0.7), scaling agent count will not raise the accuracy ceiling because new agents fail in the same ways as existing ones. The bottleneck is aggregation, not pool size. Fix the aggregator first.
EvidenceThe Beyond Consensus paper showed that adding agents and diversifying prompts left error correlations across agents essentially unchanged. The majority voting ceiling is a structural property of aggregating at the final-answer level, not a symptom of insufficient agent diversity. Computing pairwise correlations on a small test set before a scale-out experiment distinguishes between the two root causes cheaply.
-
WhenYou are red-teaming an AI agent’s resistance to prompt injection and want to identify gaps that synthetic jailbreak strings would miss.
UseBuild the attack suite from the agent’s own task context. Generate injections written to resemble the kinds of content the agent normally processes: if the agent reads legal documents, craft injections that look like legitimate contract language; if the agent handles email, craft injections that resemble routine user requests. Test whether the agent responds differently to injections framed as contextually normal content versus syntactically obvious adversarial strings. Defenses that fail context-aware attacks but pass synthetic-string tests have a known residual risk that should be documented and monitored.
EvidenceAbdelnabi and Bagdasarian (2026) apply Contextual Integrity theory to prompt injection and derive an informal impossibility result: for any norm a defender chooses, an attacker can construct a context in which the blocked flow appears to conform to that norm. This means the attack class that is hardest to block consists of injections that look like contextually appropriate task flows, not syntactically flagged adversarial strings. Evaluations that only test the latter class will overstate how safe the system is.
-
WhenYou maintain an agent eval suite and the suite includes only tasks the agent is designed to complete, so your primary metric is task success rate.
UseA held-out infeasibility split alongside your standard task suite. Write 10 to 15 tasks that are genuinely impossible for your agent: contradictory requirements, capabilities explicitly out of scope, resources that do not exist. Run your agent on them separately and measure the false-positive completion rate, how often it attempts rather than refuses. Treat a false-positive completion rate above 5% as a calibration problem and report both metrics in every eval cycle.
EvidenceAPB tested 12 multimodal LLMs on tasks designed to be unsolvable across 22 domains. Models showed systematic over-confidence, rarely refusing even explicitly infeasible tasks. Because standard benchmarks include only solvable tasks, this failure is invisible to teams using task-completion rate alone. The omission matters in production: an agent that always attempts an impossible task wastes its full token budget and delays the user’s recognition that the goal cannot be reached.
-
WhenYou are diagnosing agent task failures and deciding whether to invest in prompting improvements, scaffolding changes, tool reliability fixes, or model upgrades.
UseA two-category classification pass over recent failures before committing to any remediation. For each failure, ask: was the plan wrong before any tool was called, or did a correct plan break during execution? Label each failure accordingly. A bad-plan failure targets the planning context, prompting, or reasoning architecture. A bad-execution failure targets tool reliability, schema validation, error recovery, or retry logic. Count the proportion in each category and let it guide where you invest next. Without this classification, improvements to the wrong layer produce no gain.
EvidenceAPB’s core structural contribution is separating planning failure from execution failure through five distinct benchmark settings: holistic planning, feedback-conditioned step-wise planning, robustness under extraneous tools, robustness under broken tools, and infeasibility detection. Across 12 MLLMs, models showed distinct performance profiles across settings, with different models failing at different layers. A combined end-to-end pass/fail score merges these distinct failure signatures into a single number that cannot direct remediation.
-
WhenYou anticipate rotating, upgrading, or A/B testing an LLM agent in a production multi-turn environment and want to evaluate candidate replacements before live exposure.
UseInstrument structured interaction logging on the current production agent before any changes. For every multi-turn interaction, capture: conversation state before the agent’s turn, agent action, next environment turn, and task outcome. Use a format queryable by time window and agent version. Without historical trajectory data, off-policy evaluation is not possible; the time to instrument is before a rotation is needed, not during.
EvidenceADWM and all off-policy evaluation methods require pre-collected trajectory logs from the same environment. The method produces accurate value estimates and reliable candidate rankings from historical data, enabling pre-deployment screening. The quality of estimates depends directly on how well the logged data covers the states a candidate agent will encounter.
-
WhenYou are using an off-policy evaluation method to estimate how a candidate LLM agent would perform in a production environment before deploying it.
UseValidate the off-policy estimate against a small held-out live A/B slice before using it to gate a deployment decision. Treat OPE as a first-pass filter that eliminates clearly underperforming candidates; treat the top finalists as requiring live validation. This is especially important when the candidate agent’s behavior differs substantially from the agent that generated the historical logs.
EvidenceADWM’s estimate accuracy degrades under distribution shift: if the candidate policy explores states not covered in the historical data, the world model extrapolates beyond its training distribution. The paper recommends using OPE for pre-screening and candidate ranking, with a live validation step for the top finalists before full deployment.
-
WhenYou are evaluating an autonomous coding or deployment agent’s output and need to decide whether a task was completed successfully.
UseGrade agent output by clean-environment execution, not diff plausibility. Spin up a fresh environment, apply the agent’s changes, and run the artifact end-to-end. Reject any evaluation signal that only checks whether code was written or whether tests pass in the same environment the agent modified. This applies to both automated pipelines and human code review.
EvidenceDeployBench ran 51 artifact deployment tasks across AI/ML, computer systems, and scientific computing using four frontier models via OpenHands. Best pass rate was 51.0%; worst was 7.8%. 97 of 154 analyzed failures were completion-judgment errors: the agent stopped before verifying whether the artifact actually ran. Code that passed local tests did not run in clean environments. The benchmark isolates deployment success from code generation quality, revealing a systematic failure mode that per-file diffs and unit test suites cannot detect.
-
WhenYou are selecting between two or more LLM agents or foundation models for a production use case and are using published benchmark scores to inform the decision.
UseRe-run the candidate models through your own in-house harness before finalizing the selection. Define your prompt format, loop structure (ReAct, direct, chain-of-thought), tool schema, and environment setup. Run every candidate through this single configuration on a representative task sample. Use published benchmarks only for discovery and shortlisting; use your internal harness scores for the decision. Document your harness configuration alongside your results so the comparison is reproducible.
EvidenceAcross 7 agent benchmarks, 15 models, 400K rollouts, and 5B tokens, Zhu et al. (2026) found that scaffold choice and environment volatility shift outcomes in both directions, enough to reorder leaderboards. A model that ranks first on a benchmark using that benchmark’s native scaffold may rank differently under a different harness. The effect is bidirectional and cannot be predicted from benchmark scores alone. The study introduces a unified instruction-tool-environment harness and fixed ReAct loop as the controlled baseline, demonstrating that a single harness applied across all benchmarks produces different and more interpretable rankings than same-model scores across benchmark-specific scaffolds.
-
WhenYou are running agent evaluation in an environment that involves live APIs, web services, or other external systems that may change between evaluation runs.
UseMeasure environment volatility before using evaluation scores in any decision. Run the same model on the same evaluation tasks at least three times across different days without changing any prompt or model configuration. If scores vary by more than a few percentage points, the environment is contributing noise. Switch to snapshotted or cached environments for evaluation and use live environments only for final validation on the shortlisted model. Report the variance alongside the mean score so downstream decision-makers can assess reliability.
EvidenceZhu et al. (2026) introduce an offline snapshot mode that replaces volatile live environments with stable curated snapshots, enabling isolation of environment effects from model effects. Applied across 7 benchmarks, the mode reveals that environment volatility is a measurable, non-trivial contributor to score variance. Teams evaluating agents in live environments routinely conflate this noise with model performance differences, leading to selection decisions that may not hold in deployment.
-
WhenYou are evaluating a skill or tool retrieval system for an LLM agent and currently measuring only per-item retrieval metrics such as Hit@1 or MRR.
UseAdd a Set-Compat metric to your evaluation suite. Set-Compat measures whether the entire ground-truth skill set required to complete a task appears simultaneously in the top-m retrieved results -- not merely whether any one correct skill was returned. Run both per-item and set-level metrics on a held-out task split; the delta between them is the proportion of failures attributable to incompatible skill combinations rather than missed individual skills.
EvidenceOn the R3-Skill benchmark (5,696 queries, 2,050 skills), even when a bi-encoder embedding model retrieved the most individually relevant skills, the full compatible set was present in the top-20 only 22.2% of the time. A compatibility-aware reranker raised Set-Compat to 35.3% (+13.07pp) while Hit@1 improved only +6.9pp -- confirming that compatibility is a measurably distinct failure mode that per-item metrics cannot detect.
-
WhenYou are evaluating a conversational diagnostic or troubleshooting agent and your current metrics measure only final answer accuracy on clear-cut test cases where the correct root cause is unambiguous.
UseAdd a misleading-premise split to your evaluation set. Include cases where a user's initial description contains a plausible but incorrect explanation already embedded in the framing. Measure final-answer accuracy separately on these misleading-premise cases versus unambiguous cases. A model that scores well overall but drops sharply on the misleading-premise split is vulnerable to diagnostic sycophancy: it reinforces the user's hypothesis instead of generating and testing alternatives.
EvidenceThe LLM-as-an-Investigator paper identified user-driven sycophancy as the core failure mode in interactive problem diagnosis: when an assistant absorbs a user's explanation rather than independently evaluating it, the model becomes part of the misdiagnosis. Evaluating only on clear-cut cases hides this failure entirely. A benchmark split over solved mechanical, electrical, and hydraulic forum threads demonstrated that the gap between misleading and unambiguous cases is large enough to distinguish evidence-first agents from direct-answer baselines.
-
WhenYou are evaluating an LLM agent and currently report only task accuracy on tasks the agent completes, with no separate metric for how much budget is consumed on tasks the agent fails.
UseLog token usage by outcome for every agent run: succeeded, failed, and abandoned. Calculate the mean and distribution of tokens consumed on failing tasks as a fraction of the task budget. Report this wastage rate alongside accuracy. A wastage rate above 30% is a flag that budget-awareness training or architectural intervention is warranted.
EvidenceLin, Wang et al. (2026) measured wastage rate across frontier models on a standardized task suite and found ranges of 28% to 64% of the task budget consumed on ultimately failing tasks. Even the best-performing frontier model in the study wasted more than a quarter of its budget on tasks it could not complete. Wastage rate was not visible in standard accuracy-on-completion evaluations; it required a separate outcome-stratified instrumentation pass.
-
WhenYou are evaluating a tool-using agent and your benchmark uses the same tool names, query phrasings, and observation schemas that were present during training.
UseAdd at least one distributional shift split to the benchmark before treating results as deployment-ready. Start with action shift (the cheapest to implement): copy your eval set, rename each tool by substituting a synonym or version label, and run the agent on both the original and renamed-tool versions. Record the performance delta. A delta above 10 percentage points indicates high fragility to the kind of API naming changes that happen routinely in production. Repeat for query shift (paraphrase queries) and observation shift (inject noise or field reordering into tool return values).
EvidenceLv, Wu, Zhu, Cheng & Guo (2026, ICML) showed that both SFT-trained and RL-trained tool-using agents degrade significantly under each of four shift types -- query, action, observation, and domain -- even when the underlying task and intent are unchanged. Action shift (renamed tools) produced the most consistent drops because agents trained on specific tool vocabularies fail to identify the correct tool when its surface label changes. Standard held-out eval sets use the same tool vocabulary as training, making them closed circuits that measure memorization rather than adaptation.
-
WhenYou are evaluating two or more agent frameworks or base models on the same task set and want to know whether a performance difference reflects the model’s capability or the framework’s design choices.
UseRun the same base model under at least two different agent frameworks on a shared representative task set before drawing any conclusions about model quality. Measure per-capability scores (skill usage, exploration, long-context reasoning) separately rather than relying on a composite score. A gap of more than a few percentage points across frameworks on the same model signals a framework effect: the model is not the variable you think it is, and changing the model without understanding the framework will not fix the problem.
EvidenceUniClawBench ran identical base models under different agent frameworks on 400 bilingual real-world tasks graded via live Docker containers. Capability scores diverged meaningfully across frameworks for the same model, with the direction and magnitude of divergence varying by capability dimension. The benchmark was designed specifically to disentangle model effects from framework effects after observing that composite scores on static benchmarks systematically conflated the two.
-
WhenYou are designing or improving an internal eval pipeline for multi-step LLM agents and currently grade only on final task output.
UseDecompose each task into explicit intermediate checkpoints before evaluation begins. Define the success criterion for each checkpoint in advance. Use a grading mechanism that scores progress at each checkpoint rather than only at the final answer. This surfaces two distinct failure modes a final-answer eval cannot distinguish: agents that fail immediately (never started) versus agents that stall mid-task after completing early steps correctly.
EvidenceUniClawBench’s checkpoint grading design, applied to 400 real-world agent tasks in live Docker environments, identified mid-task stall as a distinct and common failure pattern. Some agents scoring zero on final-answer matching had completed the first several checkpoints correctly. That partial-progress signal is actionable: it indicates the agent can initiate the task but cannot sustain performance, which requires different remediation than an agent that fails from the first step.
-
WhenYou are evaluating an agent’s accuracy on a structured task domain and need to understand whether poor performance reflects reasoning failures or state-validity failures.
UseSample completed tasks from the agent’s history and programmatically verify the resulting state against the policies the agent is supposed to enforce. Record the fraction that the agent rated successful but left the system in a policy-violating state. Report this silent-failure rate separately from reasoning-error rate, tool-rejection rate, and task-refusal rate. The split determines whether prompt improvements, model upgrades, or post-action validation is the right lever.
EvidenceOn tau-bench airline, 78% of a budget agent’s failures were silent wrong-state writes where neither the tool nor the agent’s self-report indicated a problem. This means a large majority of failures were invisible to standard task-completion and tool-error metrics. The silent-failure rate is structurally distinct from other failure modes and responds to different interventions: deterministic state validation rather than reasoning improvements. REDAgentBench (Chen, Liu, Zhu, Dou et al. 2026) further confirmed the principle across banking, healthcare, and e-commerce agent scenarios, finding that trajectory-only evaluation missed 18.4% of confirmed violations.
-
WhenYou are running A/B tests or model comparisons in a production system where evaluation is expensive per trial and you want to stop as soon as you have a statistically valid conclusion rather than at a predetermined fixed horizon.
UseReplace the fixed-horizon hypothesis test in your evaluation pipeline with an anytime-valid equivalent (e-values or sequential likelihood ratio tests). Set a false-positive budget (alpha) before the evaluation begins. Monitor the test statistic continuously; stop and record a conclusion as soon as the budget is spent or the stopping criterion is met. Log the stopping step alongside the result so reviewers can distinguish early-stop conclusions from full-horizon conclusions. Run one evaluation using the old fixed-horizon test and one using the anytime-valid test on the same data to calibrate how often early stopping would have changed your decision.
EvidenceSEA (Sengupta 2026) uses anytime-valid statistical gates as the core admission mechanism for agent self-modification. Classical fixed-horizon tests control the false-positive rate only at the pre-specified endpoint, while anytime-valid tests control it at every possible stopping time. This property allows evaluation to terminate as soon as sufficient evidence accumulates, avoiding both premature commitment at a fixed horizon and the multiple-testing inflation that comes from checking significance repeatedly. The gate produced auditable certificates covering the evaluation trajectory, the stopping point, and the test statistic.
-
WhenYou are evaluating a memory-write defense (keyword filter, consensus-deviation check, or structural screening tool) before deploying it in a production agent that relies on persistent memory.
UseMeasure the defense’s false positive rate on a representative set of authentic agent outputs before trusting the attack success reduction it reports. Run the defense against at least 200 to 300 benign traces drawn from the same agent and task distribution you plan to protect. If the false positive rate exceeds 1%, treat the defense as not yet production-ready and tune the threshold before deployment. Report both the attack success reduction and the false positive rate together: a defense metric without both numbers is incomplete.
EvidenceSENTINEL’s Reasoning Guard reduced FARMA’s attack success rate to as low as 0% while producing zero false positives across 326 benign agent traces. The paper explicitly reports both numbers with negative controls, which is unusual in the memory-defense literature. Without the false positive result, the attack suppression number alone cannot distinguish a useful defense from an overly aggressive filter that also blocks legitimate agent reasoning. The two numbers together constitute the minimum evidence for a production decision. (Karamchandani, Nagasubramaniam, Zhu & Wu, Pennsylvania State University, 2026.)
-
WhenYou are evaluating an LLM agent intended for production use in environments where inputs, documents, or outputs may appear in more than one language, and your current evaluation suite uses only monolingual (English) tasks.
UseAdd a multilingual workflow variant to at least one existing internal eval task before declaring the agent ready for multilingual deployment. Construct the variant so that the task instruction arrives in one language, the source document or reference data is in a second language, and the required output format or target is in a third. Run the same agent, unchanged, on both the monolingual and multilingual variants. Measure the pass-rate delta. If the delta exceeds 10 percentage points, treat multilingual robustness as an open engineering problem and address it before deployment rather than after.
EvidencePolyWorkBench (67 tasks, five domains, 10 languages, mean 8.5 steps per task) found that SOTA agents degrade sharply on multilingual workflows compared to monolingual counterparts, and that the failure mode is structural: a linguistic error at any step changes what is passed to every subsequent step, producing cascading degradation that single-step translation evals cannot detect. Agents that handle isolated translation tasks correctly still fail on PolyWorkBench tasks because the failure is a workflow property, not a per-step property. 88% of realistic workplace tasks examined required agents to work in three or more languages within a single workflow. (Li, Liu, Zhang et al., Beijing Jiaotong University & Tencent Weixin AI, 2026.)
-
WhenYou are using an LLM as a semantic judge to grade outputs of a multi-step agent workflow, and you are treating the judge’s scores as the primary grade rather than as a supplement alongside deterministic checks.
UseValidate the LLM judge against a structural correctness metric before trusting it as a primary grade. Run both the LLM judge and a deterministic rubric (executable verification, schema checks, or a weighted step rubric) on a representative held-out set. Compute the correlation between the two scores. If the correlation is below 0.5, treat the LLM judge as an annotation layer for failure modes that deterministic checks miss, not as the primary evaluation signal. Report both scores separately in any evaluation summary; a single combined number hides which failure modes were caught by which method.
EvidencePolyWorkBench found that the LLM semantic judge’s scores correlated with structural correctness at r=0.18 across all tasks. Within the regime where the judge expressed high confidence in its assessment, that correlation dropped to r=-0.04: a more certain judge was no more accurate than an uncertain one. The judge was useful for catching semantic consistency failures that Pytest-based verification missed, but it was not a reliable primary grade. The paper recommends pairing executable verification and weighted structural rubrics with the LLM judge, not substituting the judge for them. (Li, Liu, Zhang et al., Beijing Jiaotong University & Tencent Weixin AI, 2026.)
-
WhenYou are evaluating the efficiency of a coding or retrieval agent and currently track accuracy (task success rate) as the primary metric.
UseAdd cost per solved task as a co-primary metric alongside accuracy. For each agent invocation, record billed tokens or provider cost alongside the task outcome. Aggregate by difficulty band (simple, medium, complex) if you have a difficulty classifier, or by task category if you do not. A task that succeeds at 10x the token cost of the minimum-viable path is not performing better than one that succeeds at minimum cost. Report cost per solved task in any benchmark comparison, not cost per token or total cost alone, since the denominator (solved tasks) is what the agent is paid to produce. Use the metric to identify which task categories drive disproportionate spend before tuning prompts or adding memory.
EvidenceYin & Feng (2026) found that full-effort agent baselines spent 5-10x more compute than MSE-Bench tasks required without improving accuracy over a minimum-scope baseline. The finding was only visible by measuring cost per solved task: raw accuracy was at ceiling (100%) for both baselines and E3, masking the cost differential. The paper also corroborated token savings on a live gpt-4o harness to confirm real provider billing tracked the token-count reduction, addressing the concern raised by separate work that token-count and billed cost can diverge under caching and retry effects.
-
WhenYou are evaluating a memory-equipped LLM agent and your current metrics measure retrieval quality (Recall@K, exact match on retrieved facts) but do not separately measure whether the agent defers to retrieved memories appropriately.
UseAdd a judgment-layer eval split alongside your retrieval metrics. For each evaluation scenario, record whether the retrieval succeeded (right memory was found) and separately whether the agent used that memory appropriately (deferred when it should, pushed back when it should not). Add at least two adversarial cases: one where the stored memory contains a verifiable factual error and the agent should correct the user, and one where a memory is valid in one domain but does not transfer to the current context. Score retrieval accuracy and usage accuracy independently. A memory system that scores well on retrieval but poorly on usage judgment is a reliability risk, even if aggregate accuracy looks acceptable.
EvidenceMemSyco-Bench (Xiang, Chen, Tang et al., 2026) introduced five task categories separating appropriate from inappropriate memory deference. The benchmark found that existing memory systems often failed on judgment tasks while succeeding on retrieval, and that standard evals conflating the two masked this distinction entirely. Most failures were judgment errors: the agent retrieved the correct memory, then used it in a context where the evidence in the current conversation should have overridden the stored claim.
-
WhenYou are evaluating context management strategies for a long-horizon agent and your current benchmark measures token reduction or raw context size rather than task outcomes under a resource constraint.
UseAdd task completion rate under a fixed peak context budget as a primary evaluation metric for any context management intervention. Hold the maximum context window constant across all conditions being compared: truncation baseline, static summarizer, trained compaction. Measure how many tasks each approach completes within that fixed budget. A strategy that compresses context aggressively but loses critical information will score lower on task completion than one that compresses less but preserves what the agent needs. Separately track what percentage of post-compaction steps reference information that was in the pre-compaction context but not in the summary; this instrumentation identifies which task types are most sensitive to summary quality.
EvidenceCompactionRL’s evaluation held peak context size constant across all conditions and reported task completion rate rather than token counts or compression ratios. This framing distinguished between two different ways to handle context pressure: expanding the budget (scale) versus using the existing budget more effectively (efficiency). Both SWE-bench Verified (+7.0pp) and Terminal-Bench 2.0 (+3.1pp) gains occurred within the fixed constraint. Ablation analysis further isolated which training components drove the improvement, making the efficiency gain attributable rather than opaque. (Li, Hou, Jing, Tang & Dong, Tsinghua University, 2026.)
-
WhenYou are selecting or reporting evaluation metrics for an LLM agent that will replace or augment human labor on knowledge-work tasks.
UseAdd quality-per-dollar as a co-primary metric alongside task completion rate. Tag each benchmark task with a human labor-time estimate and a corresponding cost proxy, then compute: (agent task completion quality / agent cost) vs. (human task completion quality / human cost). Report both axes. A low-cost agent that does not produce deliverable-quality output is not a substitute for human labor; it is a cheaper way to produce unusable output. Stop treating speed and cost alone as proxies for readiness.
EvidenceZhou, Zhao et al. (2026) attached human labor-time and price-proxy labels to each task in a long-horizon office benchmark. Every model tested in their panel was faster and cheaper than a human worker on the cost axis. None produced deliverable-quality output on the quality axis. Without the economic grounding, a pass/fail score on each task would record every model as cost-competitive and leave the quality gap invisible.
-
WhenYou are building an internal benchmark to measure whether an AI agent is ready for unsupervised deployment on office or knowledge-work tasks.
UseInclude a human-labor baseline alongside the agent-vs-agent comparisons. For each benchmark task, record: the time a competent human takes to produce a deliverable-quality output, the cost of that labor, and what “deliverable quality” means for that task. Then run the same task with the agent and score quality on the same rubric used to grade the human output. An agent that is cheaper but produces undeliverable output has not cleared the bar for autonomous deployment, regardless of how it compares to other agents.
EvidenceOmegaUse-OfficeVal used economic grounding to reveal a result invisible to agent-vs-agent evaluation: all tested models were faster and cheaper than human workers on the input side, yet none reached the human output quality required for a deliverable. Agent-to-agent comparisons would have suggested the models were competitive; the human baseline showed the quality gap had not closed. The finding held across all models in the test panel. (Zhou, Zhao et al. 2026.)
-
WhenLong-context retrieval accuracy is degrading as document count grows and you are unsure whether to expand the context window, compress documents, or switch retrieval strategy.
UseRun an attention-dilution diagnostic before spending effort on window expansion or compression. For a sample of queries, measure the normalized attention mass on the gold document at several distractor counts (e.g., 10, 50, 200, 1000). If attention mass drops monotonically while per-query accuracy stays high in low-distractor conditions, the cause is softmax denominator inflation, not insufficient window size. Route to a length-aware attention fix or a dedicated small retriever rather than scaling the context window further.
EvidenceGollapudi, Gupta, Singhal, Min et al. (2026) showed that in-context retrieval collapses at million-token scale not because models lack capacity but because the softmax denominator inflates linearly with document count, draining normalized probability mass from the gold document. Their diagnostic revealed a monotonic attention-mass decline that correlated directly with retrieval failure, distinguishing dilution from a context-length limitation. Interventions targeting dilution (length-aware adjusted softmax, document-level sparse attention) restored accuracy without requiring a larger window.
-
WhenYou are A/B testing prompt variants for a reasoning-model coding agent and want to know whether one prompt is genuinely cheaper than another.
UseLog deliberation tokens and output tokens separately for each run, using the model provider’s response metadata. Report correctness (pass/fail on verifiable answers) alongside each token count. The metric for comparing prompt variants is deliberation tokens per correctly-solved task, not total tokens and not aggregate cost. Do not mix in infrastructure changes (caching configuration, context-compression layers, tool-call pruning) in the same test window; those interact with token counts in ways that obscure prompt effects.
EvidenceThe same authors’ prior study (Token Reduction Is Not Cost Reduction, arXiv:2607.12161) found that token reduction and billed cost diverge once prompt-cache traffic is counted, because cache hits and misses affect different billing buckets. Keeping deliberation tokens as the primary signal for prompt-variant comparisons avoids this confound and produces a cleaner read on whether the phrasing change actually removed waste.
-
WhenYou are running context-length ablations to measure how model accuracy changes as prompt length decreases, using prompts shortened from their original length by removing content.
UseBefore shortening any benchmark prompt, identify which spans contain task-relevant information (facts, targets, or evidence the model needs to answer correctly) and mark them as protected. Remove content only from unprotected regions when creating shorter-context variants. Run the ablation under both standard middle-drop truncation and your distractor-aware truncation at each retention fraction, report both curves, and treat the gap between them as an estimate of the truncation-methodology artifact. Do not attribute performance differences at shorter lengths to window limits unless you have ruled out that protected content was removed.
EvidenceArjmandi (2026) ran BABILong and GraphWalks BFS at four retention fractions (100/75/50/25%) under naive middle-drop truncation and distractor-aware truncation, across Claude Haiku 4.5, Sonnet 4.6, Opus 4.7, and GPT-5.5, with MRCR v2 and Oolong as replication benchmarks. Naive truncation collapsed accuracy monotonically. Distractor-aware truncation preserved or improved accuracy at every fraction, with statistically significant gains on BABILong for Haiku 4.5 and Sonnet 4.6. The result shows that a portion of published long-context degradation curves reflects signal loss from the truncation method rather than true window-limit effects.
-
WhenYou are evaluating an agent on multi-step or long-horizon tasks and reporting a pass rate or accuracy number as the headline result.
UseRun each benchmark task at three or more independent seeds and report the standard deviation alongside the mean. Compute a consistency score: the fraction of tasks the agent solves on every seed out of the tasks it solves on at least one seed. Report mean pass rate, per-task variance, and consistency together as a three-number eval summary. Do not report a single-seed pass rate as the reliability metric for production decisions; it conflates reliable performance with lucky single-run outcomes.
EvidenceA systematic study of seven frontier models on 36 long-horizon R&D tasks found that run-to-run variance was substantial across all models, with final pass rates masking meaningfully different behavioral consistency profiles. Two agents with identical mean pass rates can have very different reliability characteristics, and single-run evals do not distinguish them. The three-axis behavioral instrumentation (Solution Framing, Execution, Feedback Control) revealed variance patterns that terminal scores smoothed over.
-
WhenYou are considering deploying an agent in a setting where it will encounter repeated similar tasks over time, and you want to know whether the agent improves with use.
UseDesign a controlled experience-reuse test before accepting any self-improvement claim. Run the same task family in two conditions: cold start (no prior context) and warm start (given a prior successful run trace). Keep all other variables constant: same task, same model, same system prompt except for the added trace. Measure pass rate, solution framing approach, and feedback response patterns under both conditions. If the warm-start condition does not show measurable behavioral change on the process axes, the agent is not reusing experience, regardless of what anecdotal runs suggest.
EvidenceControlled comparisons in the Beyond Final Scores study found that neither within-task experience (multiple attempts at the same task) nor cross-task experience (completing one R&D task before starting another) produced measurable improvement on process-level behavioral metrics across seven frontier models. Agents reset behaviorally between runs. The finding contradicts the common assumption that in-context prior runs improve agent behavior, and shows the assumption requires an explicit controlled test rather than inference from aggregate pass rates.
-
WhenYou are building or scaling a skill library for an agent system and want to track whether retrieval is becoming a bottleneck as the library grows.
UseInstrument your skill retrieval pipeline to log actual-use precision separately from end-task success rate. At each library size checkpoint (10, 25, 50, 100 skills), sample a held-out set of tasks, record which skill was retrieved, and mark it correct only if the agent used the retrieved skill in a way that contributed to task completion. Report this metric alongside task success; if actual-use precision drops significantly as the library grows while task success holds steady, the agent is compensating for retrieval failures rather than benefiting from the library. Treat the retrieval layer as a separate engineering problem from skill content quality.
EvidenceJiang, Huang, Xing et al. (2026) measured retrieval precision across pool sizes from 5 to 100 skills and found actual-use precision collapsed from 29.6% at 5 skills to 3.3% at 100 skills. Crucially, exact ground-truth skill invocation was neither sufficient nor necessary for task success, meaning offline retrieval accuracy benchmarks are a poor proxy for whether retrieval is actually helping the agent. The bottleneck is scale-sensitivity of the retriever, not the quality of the skills themselves.
-
WhenYou are comparing context-pruning configurations for an agent pipeline and currently track only token reduction to evaluate which approach wins.
UseAdd quality (answer correctness or relevance) and faithfulness (how well the answer covers the source material) alongside token reduction as tracked metrics in every pruning comparison. Report all three independently. Identify which metric your use case prioritizes before selecting a configuration, because no pruning configuration dominates on all three simultaneously.
EvidenceKolukuluru, Dernoncourt, Rossi, Lipka et al. (2026) evaluated six pruning configurations (three positions, two scoring methods) across quality, efficiency, and faithfulness and found no single configuration dominated all three. The configuration that maximized token reduction was not the one that maximized faithfulness. This is an honest negative result the cost-aware literature typically elides. Reporting only token reduction can mask faithfulness degradation that would matter in production.
-
WhenYou are evaluating a tool-retrieval or skill-retrieval system using a benchmark and want to know whether the benchmark performance predicts production performance.
UseCheck whether the benchmark’s test queries come from the same source styles as your production queries and as the retriever’s training data. If the test and training queries share a source style, treat the published coverage or top-1 number as an upper bound on single-source performance rather than a production forecast. Add a per-source coverage breakdown to your internal evaluation before selecting or deploying a retriever, and report coverage separately for each source type in your production stream.
EvidenceLiu, James, Wang, Xiao & Lin (2026) found that a retriever performing at 86.1% coverage on its training source style scored 22.3% on a mixed stream, a 64-point collapse invisible to aggregate task-success metrics. Benchmarks that evaluate on the same source style used for training systematically overstate generalization. The coverage-weighted global top-1 proxy without routing was 1.3% across five collapsed sources, near chance, while single-source benchmark numbers would not predict this.
-
WhenYou are evaluating a fine-tuning method (self-training, LoRA, or similar) by comparing the trained model’s test scores to a pre-training baseline and want to know whether the observed gains are real.
UseRun the frozen, unmodified base model through the same evaluation pipeline as the trained model, multiple times. Collect the per-problem score distribution from these null rollouts. Compute improvement on the trained model using a per-problem exact test against the pooled null baseline, with false-discovery-rate correction applied across the full problem set. Any per-problem gain that does not exceed the null distribution is a measurement artifact, not a capability change. Report the null distribution alongside the trained-model results.
EvidenceXu, Yan, Chen & Kechadi (2026) ran a frozen Qwen3-8B through the standard self-improvement evaluation pipeline and identified seven measurement failures, each of which inverts a reported finding when the null is present. A single greedy decode baseline manufactures apparent capability changes in the untrained model via inference-batching artifacts. Under per-problem exact tests with FDR correction, three forms of self-training showed no statistically detectable improvement on any held-out replicate. Code and evaluation artifacts are released alongside the paper.
-
WhenYou are benchmarking agent systems across multiple domains with incommensurable outcome metrics (e.g., reinforcement learning accuracy alongside segmentation F1 alongside optimization loss).
UseNormalize all results to an anchored scale: 0 for an uninformative baseline, 0.1 (or another calibrated value) for the existing shipping method, and 1.0 for the theoretical task optimum. Report progress as position on that scale rather than raw metric values. This makes cross-domain leaderboards readable without domain expertise for each entry.
EvidenceAI4AI-Bench applied this normalization across 10 training algorithm families with incommensurable metrics. The scale made aggregate comparisons across 29 configurations and 6 systems legible and surfaced the mechanism-vs-parameter behavioral split that raw scores would have obscured. Chi, Li, Hong, Wang et al. (2026).
-
WhenYou are considering rebuilding an agent training or evaluation environment because it no longer challenges the current model effectively.
UseTry wrapping the existing environment before rebuilding it. Instrument failure trajectories in the current environment, synthesize targeted harness modifications for the top two or three failure clusters, and validate on a held-out set. If the wrapped version matches what you expect from a rebuild on held-out performance and execution efficiency, the rebuild is unnecessary. Report held-out performance separately from in-distribution performance; a gain only on in-distribution tasks is a sign the harness overfit to its construction conditions, not a real improvement.
EvidenceThe same EnvHarness study found that the verifier is the most expensive and fragile component of any environment to replace. Wrapping preserves the original verifier while reshaping behavior, separating the question of what scenarios to present from the question of whether the agent succeeded. Gains on held-out instances in the study suggest the harness pushed the agent toward more general capabilities rather than task-specific patches.
-
WhenYou are selecting or comparing LLM models using scores from a published leaderboard (ARC, HellaSwag, MMLU, TruthfulQA, or similar multiple-choice benchmarks).
UseCheck the scoring mode the leaderboard used before relying on the ranking. Determine whether the harness scored by generated text or by per-option log-likelihood. If the documentation does not specify, treat the score as provisional. For decisions between closely-ranked models, run the same benchmark under both scoring modes on your own evaluation set and report the range rather than a single number.
EvidenceAcross 12 open-weight instruction-tuned models evaluated on 3,679 items from ARC, HellaSwag, MMLU, and TruthfulQA under 26 equally defensible harness configurations, scoring mode (generated text vs. per-option log-likelihood) was the load-bearing variable driving harness-induced score variance. One model spanned 31% to 89% accuracy across configurations. Four of twelve models could claim rank one under some configuration. Config-fragile items carried 95.7% of the gap between adjacent model pairs on average; on items where adjacent models both answered stably, they scored essentially the same.
-
WhenYou are compressing a benchmark by selecting a subset of high-discrimination items, or you are using a published compressed benchmark for model selection.
UseMeasure the fragility profile of the benchmark before relying on compressed item sets. Run two defensible configurations (at minimum: one generated-text scoring run and one per-option log-likelihood run) and check whether rankings change. A benchmark that reverses rankings on scoring-mode change is not safe for model selection decisions regardless of its discrimination profile. If you are assembling a compressed benchmark, prefer stable high-discrimination items over fragile ones.
EvidenceItem discrimination, the property that benchmark compression methods maximize, correlates with config-fragility at r = 0.28 (95% CI: 0.25-0.30) across 3,679 items from four standard benchmarks. Selecting for discrimination is, on average, selecting for fragility. A benchmark compressed by standard discrimination criteria is more likely to be fragility-weighted than the full benchmark it was derived from. The analysis script and per-item fragility records were released with the paper and run on a CPU in seconds.
-
WhenYou are evaluating an agent system that uses a tool set with three or more tools and you need a diagnostic beyond aggregate task success.
UseSeed your staging tool set with at least one canary per tool scope boundary: a tool that looks applicable to your common task types but requires a permission, context, or prerequisite the agent cannot satisfy. Measure canary susceptibility rate (CSR) as the fraction of tasks where the agent selects at least one canary. Track CSR as a standing metric alongside task success, and re-run when your tool set expands or your model version changes.
EvidenceAcross 8,640 runs on eight models at three canary-density levels and three seeds, canary susceptibility rate varied by a factor of 36 across model families. Aggregate task success did not reveal these differences. CSR correlated with downstream task failure at Spearman rho = -0.34, making it a predictive signal of production reliability separate from pass rate. A subtlety ablation (softened canary phrasing) left frontier CSR essentially unchanged, confirming the metric measures reasoning rather than word-matching.
-
WhenYou are accepting agent output on a whole-repository migration, refactor, or stack change and deciding whether to merge it.
UseAdd a migration-completeness check before running the test suite. Verify that the migrated repository does not contain the original implementation preserved under a different path or namespace. A diff of file structure and key import patterns is enough to catch the most common form of completeness failure. Run this check first: if it fails, the test results are meaningless.
EvidenceSWE Refactor Bench found that coding agents can pass behavioural test suites without completing the migration, a failure mode the authors call Blindness. Among 520 runs from 8 frontier models, only 28 (5.4%) cleared all three evaluation stages, including a Migration Audit that detects this failure before tests run. 13 of 20 migration tasks received zero complete solutions. Adding the completeness gate is cheap relative to discovering in production that the agent reproduced the original code.
-
WhenYou are setting a merge gate on agent-produced migration or refactor work and deciding how to interpret the test-pass rate.
UseSet the merge gate at 100% of the fixed test suite on whole-repository migration work. Do not treat 99% as a near-pass. Supplement the fixed suite with agent-generated targeted tests before merging: commission a separate coding agent to probe for behavioural differences between the submitted and original code. Report test-pass rate and completeness separately rather than as a combined score.
EvidenceAmong the 340 SWE Refactor Bench runs that passed the Migration Audit, 58% reached 99% of fixed-test coverage while only 26% reached 100%. That gap, from 99% to the final point, is the difference between a result that would be accepted under partial credit and one that would fail a merge gate. The benchmark also found that migration completeness and behavioural correctness are separate abilities: passing tests does not imply the migration was complete, and completing the migration does not guarantee no regressions.
-
WhenYou are evaluating any change to a model's attention mechanism intended to reduce memory or inference cost, including window-size changes, attention sparsity, or post-trained alternatives to full attention.
UseAdd a long-context reasoning benchmark to the evaluation. Measure retrieval accuracy at multiple context lengths using Needle-in-a-Haystack or a comparable task. Report long-context reasoning quality alongside memory and latency metrics. Do not report an efficiency intervention as successful on memory or speed savings alone if long-context reasoning quality has degraded relative to a training-free baseline.
EvidenceIn Jolicoeur-Martineau et al. (2026), the 2 to 10 times performance gap between SWA and post-trained linear attention was visible only on long-context reasoning benchmarks, not on general downstream tasks. Evaluations that omit this dimension would incorrectly conclude that post-trained linear attention matches or beats SWA, which is the conclusion the prior literature reached.