Architecture
How the system is shaped: agent topology, planning, step-level evaluation, and the structural choices that determine whether compute is spent well.
No practices in this stage match the current filters.
-
WhenYou’re choosing an architecture for a multi-hop reasoning task and the input context is well-curated and not adversarial.
UseA single agent with a generous thinking-token budget as the default. Reach for multi-agent orchestration only when the input is noisy, adversarial, or too long for the model to use effectively. The crossover where multi-agent earns its overhead happens at heavy context degradation, not at moderate task complexity.
EvidenceAcross three model families (Qwen3, DeepSeek, Gemini 2.5), five multi-agent architectures, and two multi-hop reasoning benchmarks (FRAMES and MuSiQue 4-hop), single-agent systems matched or beat every multi-agent variant under matched thinking-token budgets. Multi-agent variants only pulled ahead under heavy context corruption, such as 70% token substitution.
-
WhenYou’re building an agent that runs multiple tool calls (search, retrieval, code execution, API requests) per task and want to avoid wasting budget on dead-end paths.
UseA critic that grades every step as it happens, not just the final answer. Score the marginal gain from each step rather than the absolute quality of the trajectory. Catching a wrong turn after one tool call is far cheaper than discovering it after ten.
EvidenceBudget-Aware Value Trees outperformed standard tool-augmented agents on four multi-hop QA benchmarks across two model families. The technique at 5 tool calls reached 33.8% accuracy, beating standard agents at 20 tool calls (33.4%). Step-level scoring of marginal gain was more reliable than absolute self-assessment, which LLMs are known to inflate.
-
WhenYou’re starting any task that requires multiple tool calls, retrieval steps, or LLM iterations, whether that’s an autonomous agent or a vibe-coding session.
UseA short up-front planning pass that sketches the logical shape of the problem before spending any compute. The plan does not need facts. It just needs the abstract steps, the expected number of calls, and the dependencies. Then execute against that plan and re-plan only when reality contradicts it.
EvidenceTwo independent studies converged on this. Budget-Aware Value Trees use an explicit plan node before any tool call and outperform unplanned standard agents at 4x the compute. Separately, the vibe-coding qualitative study found “plan before you vibe” was one of the two most universal community-derived best practices, addressing the highest-severity failure modes around runaway code changes and structural breakdown.
-
WhenYou are building an AI system that gives personalized feedback, instruction, or coaching to individual users, and the pipeline currently makes a single LLM call to generate the response directly from user input.
UseSeparate the pipeline into at least two stages: a reasoning stage that infers the user’s current state (knowledge gaps, emotional state, likely misconceptions) before a response stage that generates the actual reply. Treat the inferred state as an explicit intermediate artifact, not an implicit assumption baked into the prompt.
EvidenceThe SLOW framework demonstrates that separating learner-state inference from instructional response generation produces tutoring responses rated as more personalized, emotionally sensitive, and clear than single-pass LLM generation across hybrid human-AI evaluation. Ablation studies show each reasoning stage contributes independently; none can be removed without degrading output quality.
-
WhenYou are building or evaluating a retrieval-augmented generation system and need to decide how much of the optimization budget to spend on retrieval strategy versus context length.
UseRetrieval precision as the primary lever before increasing context size. Design harness search experiments that compare selective retrieval approaches against larger context windows. More tokens are not always better: the Meta-Harness text classification result achieved a 7.7-point improvement using 4x fewer context tokens than the SOTA baseline, because the discovered harness retrieved more relevant content rather than more content.
EvidenceOn an online text classification task, a harness discovered by automated search improved over a state-of-the-art context management system by 7.7 points while using 4 times fewer context tokens. The discovered harness also generalized to 9 out-of-distribution task variants (73.1% average accuracy), indicating it captured structural improvements to retrieval rather than task-specific overfitting.
-
WhenYou are designing a multi-agent system that needs to make progress on open-ended, iterative problems where failed intermediate attempts contain useful information.
UsePersistent storage of rejected outputs as first-class artifacts. Record not just successful outputs but failed attempts alongside the identified reasons they failed. Surface this failure history to downstream agents so they do not re-explore dead ends and can build on what the failure revealed.
EvidenceIn the AI Co-Mathematician system, a reviewer agent caught a flaw in a first-pass proof attempt. The flawed proof and the specific flaw were stored. When the human collaborator saw both the rejected proof and the reviewer’s identified weakness, they recognized immediately how to close the gap. The stored failure, with its explanation, was more useful than a silent rejection would have been.
-
WhenYou are designing the memory layer for an agent that will operate across multiple sessions or long time horizons, where stored facts can become outdated as context evolves.
UseValidity as a first-class property alongside relevance in your memory architecture. Assign each stored item a freshness window based on how quickly that category of information typically changes. At retrieval time, surface freshness metadata to the model alongside the retrieved content, and include a validity check in the prompt before the model acts on retrieved information. Do not rely on the model to independently notice that retrieved content may be stale.
EvidenceSTALE demonstrates that frontier LLMs at 55.2% accuracy cannot reliably self-detect memory invalidity, even when given context that implies it. The benchmark’s third axis, implicit policy adaptation, shows the hardest failure: the model must proactively update its behavior based on an implied change it was never explicitly told about. This failure is systematic enough across frontier models that architectural mitigation is more reliable than prompting alone.
-
WhenYou are deploying an LLM agent that connects to more than 50 tools via MCP and are experiencing failures that do not appear in simpler benchmarks or internal evals.
UseAudit tool retrieval as the first failure hypothesis before investigating model reasoning or plan generation. Log which tool the agent selected for each step and whether it was the correct tool. Treat a wrong tool selection rate above 10% as a retrieval-architecture problem, not a model capability problem. Mitigations to test: hierarchical tool indexing (coarse category filter before fine-grained selection), sharper semantic differentiation in tool descriptions, and query-time disambiguation that requires the agent to confirm its selection before calling.
EvidenceLi, Yang, Wang et al. (2026) built ComplexMCP, a benchmark with 150+ interdependent stateful MCP tools across 7 domains. At that scale, tool-retrieval saturation emerged as the primary failure mode across frontier models: agents selected the wrong tool due to overlapping descriptions, causing failures that originated before any planning or execution began. The failure mode was invisible in benchmarks with smaller tool sets, where distinct tools do not compete the way 150 overlapping options do.
-
WhenYou are running a chain-of-thought reasoning workload at scale and need to reduce token costs without sacrificing accuracy, and your current approach applies explicit CoT uniformly to every query regardless of difficulty.
UseTwo-phase inference as the default evaluation pattern before committing to a fixed CoT length budget. Route queries through a latent-space exploration phase first, and switch to explicit chain-of-thought only when the latent phase signals uncertainty or reaches a confidence threshold. Measure accuracy and token count both before and after the split to confirm the tradeoff is favorable on your task distribution.
EvidenceLi, Wang, Liu et al. (2026) applied this two-phase approach (LaTER) to Qwen3-14B without any additional training. Token usage fell 32% compared to uniform explicit CoT. On AIME 2025, accuracy rose from 70.0% to 73.3% at 10,661 tokens versus 15,730 tokens. The accuracy gain alongside the cost reduction indicates that latent exploration produces better intermediate representations for hard reasoning steps rather than just truncating them. The approach requires no fine-tuning and was validated across multiple model families, indicating the gain comes from the phased structure rather than a model-specific property.
-
WhenYou are deploying an agent pipeline and have no labeled routing history, but the cost gap between direct LLM inference and full agent execution is large enough to make routing worthwhile.
UseA seed-set-based experience memory to bootstrap routing before any training data exists. Select a small set of queries that span the difficulty range you expect in production, run both the base model and the full agent on each, and record which system performed better. Use this memory to route new queries by retrieving similar past cases and applying a structured scoring step to decide whether agent capabilities are actually needed. Build the seed set to include genuinely easy queries (direct model call sufficient), borderline queries, and hard queries requiring agent execution. A seed set made only of hard queries will over-escalate.
EvidenceWang, Qiu et al. (2026) built BoundaryRouter, a training-free router using early behavioral experience and rubric-guided reasoning. Starting from a seed set only, it reduced inference time by 60.6% versus always running the agent while improving accuracy by 28.6% over always using direct LLM inference. Prompt-based routing without experience memory underperformed by 37.9 percentage points, establishing that the memory component carries most of the routing signal.
-
WhenYou are building or diagnosing an agentic retrieval-augmented generation system and are considering improving retrieval accuracy by upgrading your embedding pipeline or vector database.
UseA controlled comparison of grep vs vector retrieval inside your actual production harness before investing further in embedding infrastructure. Select 50 to 100 representative queries, run each through both retrieval paths in your specific harness and model combination, and measure accuracy. Let the result determine investment direction. Do not rely on isolated retrieval benchmark results, which are measured outside the harness and may not reflect which method your specific framework and model favor.
EvidenceSen, Kasturi, Lumer, Gulati, Subbiah et al. (2026) ran 116 LongMemEval questions through grep and vector retrieval across four agent harnesses (Chronos, Claude Code, Codex, Gemini CLI). The harness layer moved accuracy more than the choice between retrieval methods. Claude Code with Opus and Haiku showed a persistent grep advantage; Gemini CLI with Gemini 3.1 Pro showed a persistent vector advantage. Same benchmark, opposite winners, driven by harness rather than retrieval method.
-
WhenYou are designing an inference-time orchestration layer (best-of-N sampling, agent committee, self-consistency) for a task and deciding whether multiple model calls will meaningfully improve accuracy over a single call.
UseA local verifier check before building the committee. Ask whether the task has an execution-based feedback mechanism: a test suite for code, a proof checker for formal reasoning, a constraint solver for planning, a type system for synthesis. If one exists, a committee design can reliably identify which candidate answer is correct and improve accuracy over single-call baselines. If no local verifier exists, invest in improving single-call quality instead, because the committee cannot determine which candidate to select without a soundness signal.
EvidenceSunkaraneni, Beneventano, Neumarker, Poggio and Galanti (2026) proved formally that inference-time agent committees succeed only when the task provides a local soundness signal for identifiability. Empirically, a nano-model committee using SWE-bench test suites as the verifier reached 76.4% on SWE-bench Verified, matching Gemini 3 Pro and Claude Opus 4.5 Thinking standalone (up from 67.0% single-call). The oracle ceiling with a perfect verifier was 79.0%.
-
WhenYou are optimizing a test-time scaling strategy (self-consistency, chain-of-thought search) and the current thresholds for branching, probing, or stopping were set by intuition and hand-tuning experiments.
UseBuild a frozen replay cache of reasoning traces from your base model on a representative problem set, then run a coding agent to discover the controller program against that cache. Evaluate candidate controllers without live model calls during the search loop. Re-run discovery whenever you change base models or shift problem distributions. The cache-based evaluation makes each candidate assessment nearly free, so the agent can iterate far more extensively than human researchers can with live-inference experiments.
EvidenceAutoTTS (Zheng et al. 2026) discovered the Confidence Momentum Controller (CMC) by running a coding agent against a frozen AIME24 trace cache with zero LLM calls during evaluation. One complete discovery run cost $39.90 and 160 minutes. CMC reduced token usage by 69.5% compared to SC@64 (self-consistency with 64 parallel samples) while maintaining matched average accuracy across four Qwen3 model scales (45.3 vs 45.2). CMC outperformed every manually designed baseline and generalized to held-out AIME25 and HMMT25 benchmarks without retuning.
-
WhenYou need to expose a test-time scaling strategy’s cost-accuracy operating point to infrastructure or product teams who cannot interpret or modify internal controller logic.
UseA single continuous β parameter (0 = maximum token efficiency, 1 = maximum peak accuracy) that maps deterministically to all internal thresholds in the controller. Document the β-accuracy and β-token curves on representative test tasks before deployment so teams can select an operating point without running ablations. Both ends of the curve and the midpoint should be characterized; the efficiency gains are largest near β=0.5, not at the extremes.
EvidenceAutoTTS’s β-parameterization collapsed the CMC controller’s multi-threshold configuration into one knob. At β=0.5, tokens fell 69.5% vs SC@64 at matched accuracy (45.3 vs 45.2 averaged across four Qwen3 model scales). At β=1.0, CMC exceeded all handcrafted baselines in 5 of 8 model-benchmark pairs on peak accuracy. Both operating points came from the same discovered controller without re-running discovery.
-
WhenYou are running self-consistency sampling (generating N reasoning traces and aggregating by majority vote) and want to improve answer quality without generating more traces.
UseConfidence-weighted aggregation instead of simple plurality vote. Score each reasoning trace by quality indicators before counting its vote. Weight answers by the accumulated confidence of the traces that produced them. High-confidence convergence on an answer is a stronger signal than raw count alone. Implement this as the first optimization pass before adding any path-level pruning.
EvidenceDDC’s Confidence-Weighted Bayesian Voting (CWBT) component improved accuracy on AIME 2025 by 15.6 percentage points over standard self-consistency using Qwen3-4B, because quality-weighted aggregation better identifies the correct answer when trace quality varies across the sampled pool. The accuracy gain appears even before factoring in the token reduction from early termination.
-
WhenYou are building an automated interpretability pipeline and currently select which model features to examine by activation frequency rank or random sampling.
UseA co-activation graph as the primary selection mechanism before any explanation work begins. Build a k-NN graph where each node is a feature (neuron or circuit element) and edges connect features that tend to co-activate on similar inputs. Apply a statistical separability metric to each node: features with high separability have crisp, testable activation patterns; features with low separability are entangled and hard to explain precisely. Route high-separability candidates to your explanation loop first. Frequency-based or random selection wastes explanation budget on features that are too diffuse to describe accurately.
EvidenceMarin-Llobet and Ferrando (2026) demonstrated that navigating activation space via a k-NN co-activation graph with statistical separability scoring outperformed alternative feature selection strategies for mechanistic interpretability. Features identified by the graph-based discovery agent produced better explanation outcomes on Gemma-2 and weight-sparse MLP neurons than those selected by frequency or at random, because high-separability features are precisely the ones an explanation loop can test and verify.
-
WhenYou are operating an AI agent that executes real-world tool calls (shell commands, file operations, HTTP requests, database queries) and your current safety posture relies on post-hoc audit logs or static keyword filters on raw tool call strings.
UseA runtime interception layer that evaluates the semantic intent of every tool call before execution and returns a structured verdict (allow, warn, block, review). Place the interceptor between the agent and its tools so no call executes without passing through evaluation. Treat the interceptor as distinct from and complementary to sandbox restrictions: the sandbox constrains the execution environment; the interceptor evaluates what each action means inside that environment.
EvidenceYang (2026) built AgentTrust, an 8-component runtime interceptor for agent tool calls. On a 300-scenario internal benchmark spanning six risk categories (file operations, network access, code execution, credential exposure, data exfiltration, system configuration) the production ruleset achieved 95.0% verdict accuracy at low-millisecond median latency. On 630 independently constructed real-world adversarial scenarios covering DevOps, cloud, container, and supply-chain operations, verdict accuracy was 96.7%. Latency overhead was small enough to be below network call variance for typical tool execution.
-
WhenAn agent pipeline is losing track of earlier context within a single session, and RAG retrieval is too slow, too brittle, or too expensive for the session length you need.
UseEvaluate online associative memory as a complement to or replacement for retrieval in the within-session memory slot. Specifically: if the task requires the model to remember what it did or was told several thousand tokens ago but the information is not clearly chunk-able or query-able, an online state-based mechanism that updates continuously at decode time addresses a different failure mode from retrieval. Baseline on MemoryAgentBench before and after to measure actual gain.
EvidenceLei, Zhang, Li, Wang et al. (declare-lab / Nanyang Technological University, 2026) added a compact 8x8 online state matrix to frozen Qwen3-4B, Qwen3-8B, and SmolLM3-3B models via LoRA-style adapters. The state updates via delta-rule learning at inference time, with no changes to the backbone. On MemoryAgentBench it scored 1.31x the frozen backbone and 1.15x the strongest retrieval baseline; on LoCoMo (long-term conversational memory) it scored 1.20x the backbone. General-purpose benchmarks (HotpotQA, IFEval, GPQA Diamond) showed near-baseline scores, confirming the adapter did not degrade non-memory tasks.
-
WhenYou are selecting a memory architecture for a production agent and need to compare candidates on more than retrieval quality and latency.
UseAdd a safety criterion to memory architecture selection. Before committing to a design, run the trigger-probe protocol against each candidate architecture with a matched probe set and a NullMemory baseline. Compare memory-induced violation rates across architectures at equivalent exposure lengths. The architecture with the lowest rate at production-scale exposure is the safety-preferable choice, independent of capability rankings. If resources do not allow a full protocol run, at minimum instrument the retrieval layer to detect elevated similarity to unsafe content patterns before generation, as the paper shows risk is detectable at retrieval time.
EvidenceAcross the 8 memory architectures tested, not all configurations produced the same memory-induced violation rate. The paper establishes that architecture choice is a variable in longitudinal safety outcomes, not just a capability variable. The finding that risk is detectable at retrieval time before generation provides a tractable monitoring hook that architecture-level logging can expose.
-
WhenYou are choosing a global compression setting (quantization, pruning, attention sparsity) for a production LLM and have not profiled per-token difficulty in your actual workload.
UseBefore committing to a uniform compression budget, measure the distribution of output entropy across a representative sample of your production traffic. Log top-1 token probability at each decode step. If a large fraction of steps have near-certain predictions, your compression budget is over-allocated to easy tokens and you have room to recover quality on hard tokens without increasing total compute.
EvidenceAkhauri & Abdelfattah (2026) found that a learned per-token scheduling policy outperformed uniform compression by up to 7.3 MMLU points at matched FLOPs, demonstrating that token difficulty is non-uniform and that uniform budgets leave quality on the table. The gain comes entirely from reallocating the same compute toward steps the model finds genuinely uncertain.
-
WhenYour agent pipeline has a memory system and you are investigating why retrieval is returning stale or contradictory information, even after improving embedding models or chunk sizes.
UseAudit every memory write in the pipeline before touching retrieval infrastructure. Label each write by its operator intent: ingest (new information the agent did not previously know), revise (correction or update to an existing stored belief), or forget (deliberate removal). If revise operations are implemented as delete-then-insert, you are losing provenance and creating a window where retrieval returns nothing or the wrong version. Fix the data-model issue before optimizing the retrieval layer.
EvidenceOrogat and Mansour (2026) formalize long-term agent memory as Governed Evolving Memory (GEM), showing that existing vector stores support ingest and retrieve but lack a revise operator with provenance tracking. The delete-then-insert workaround teams use to simulate revision loses update history and creates retrieval ambiguity during any non-atomic execution window. The paper argues the revise-operator gap, not retrieval quality, is the primary structural cause of agent memory failures in production.
-
WhenYou are running a mixture-of-agents or self-consistency pipeline and adding more agents or more diverse prompts is not improving accuracy past a plateau.
UseReplace the majority vote or final-answer aggregation step with an LLM that reads the complete reasoning chains from all agents. Pass every agent’s full trace, not just its conclusion, to the aggregating model. Use anchored refinement so the synthesis is only accepted when it does not degrade below the majority baseline.
EvidenceFadnavis, Kanakaraj and Wyss (2026) measured error correlations across varied prompts, samplers, and agent counts and found they remain high regardless of diversity applied to inputs. The trace-reading aggregator recovered correct answers from cases where all agents agreed on the wrong final answer. Across five benchmarks, beneficial corrections by the trace reader outweighed harmful ones. The SC-MoA variant with anchored refinement provided provable non-degradation against the majority baseline and achieved highest accuracy on all five benchmarks.
-
WhenYou are choosing how to layer injection defenses in a production AI agent and deciding how much to rely on data-instruction separation as the primary control.
UseTreat data-instruction separation as one layer in a defense stack, not as the complete answer. Layer it with: explicit trust-boundary scoping (which contexts can trigger consequential actions), behavioral monitoring for anomalous action sequences that may signal a successful injection, and graceful failure behavior that surfaces suspicion rather than silently executing. Document the residual risk from context-aware attacks that separator-based controls cannot block, and use that documentation to calibrate monitoring thresholds.
EvidenceThe impossibility result in Abdelnabi and Bagdasarian (2026) shows that no single norm can simultaneously block all injections and permit all legitimate task flows. Data-instruction separation implements one such norm and is useful for blocking unsophisticated attacks, but it cannot close the gap that context-aware attacks exploit. The paper argues that graceful failure under attack is the correct target when perfect blocking is structurally unavailable.
-
WhenYou are scaling LLM inference capacity (adding GPUs, upgrading serving infrastructure, or paying for higher-throughput API tiers) because prefill latency or throughput is a bottleneck, and you have not yet measured how much of that prefill work is redundant across requests.
UseBefore adding hardware, sample a production window of prompts and measure prefix overlap across requests. Compute the fraction of tokens that appear identically in multiple requests (system prompts, retrieved documents, conversation history). If overlap is above 20%, evaluate whether a context-reuse middleware layer (reordering shared blocks to the prefix and deduplicating repeated content) can recover that capacity before committing to additional infrastructure spend.
EvidenceContextPilot, a context index, reorder, and deduplicate middleware evaluated at MLSys 2026, achieved 4 to 12x KV-cache hit rate improvements and up to 3x prefill latency reduction versus prior SOTA serving on workloads with significant cross-request context overlap, including enterprise document QA and multi-turn memory chat. Gains were workload-dependent: high-overlap workloads saw the largest returns, while low-overlap workloads saw minimal benefit.
-
WhenYou are building a RAG pipeline to supply in-context demonstrations for training or fine-tuning a reasoning model (math, code, logic), and you are selecting demonstrations by semantic or lexical similarity to the target query.
UseReplace similarity-based ranking with reasoning-utility-based ranking. Use a capable model as a judge to assess whether candidate demonstrations share transferable reasoning patterns with the target problem, not just topic overlap. Fine-tune a retriever on those utility-based labels via contrastive learning before using it to select training demonstrations.
EvidenceRA-RFT (Xiao, Ma et al., Rice/Meta 2026) found +7.1 pp on AIME 2025 avg@32 for Qwen3-1.7B and +2.8 pp for Qwen3-4B over GRPO when demonstrations are selected by reasoning-utility match instead of semantic similarity. The retrieval corpus (OpenR1-Math-220K from NuminaMath-1.5) is identical in both conditions; the only variable is how retrieval is done.
-
WhenYou are optimizing a reinforcement fine-tuning pipeline for a reasoning model and are deciding which components to improve: reward function design, training curriculum, or demonstration retrieval quality.
UseTreat retrieval quality as an independent variable from reward design and curriculum. Improvements on one axis do not interfere with the others. Develop and validate each component separately before combining. If your primary effort has been on reward shaping, upgrading to reasoning-aware retrieval is a distinct lever that can be added without revisiting the reward function or training schedule.
EvidenceXiao, Ma et al. (2026) explicitly characterize reasoning-aware retrieval as orthogonal to reward design and curricula. RA-RFT gains (+4.1 pp and +2.6 pp average across four benchmarks for 1.7B and 4B models respectively) stack on top of GRPO without requiring changes to the reward function or training schedule. The authors tested both axes independently, confirming they are non-interfering.
-
WhenYou are designing or selecting a retrieval system for routing LLM agent requests to a skill or tool library, and the agent's tasks require executing multiple skills in combination.
UseUse a two-stage retrieval architecture: a bi-encoder embedding model for fast candidate recall (top-20), followed by a cross-encoder reranker that scores query-skill pairs jointly for compatibility. Train the reranker with both standard (query, correct-skill) positive pairs and (query, rejected-skill) negative pairs derived from LLM skill-assignment decisions. Do not add rejection-based negatives to the bi-encoder training -- the structure of independent embeddings prevents them from encoding compatibility signals and the negative pairs will degrade performance.
EvidenceThe Bilateral Balancing Theorem proves formally that bi-encoders cannot use rejection or compatibility signals because the balancing gradient update cancels their effect. Ablation on R3-Skill confirmed this: adding rejection (SKIP) signals to the bi-encoder slightly hurt results, while adding them only to the cross-encoder reranker produced a +13.07pp Set-Compat gain and +6.9pp Hit@1 gain over the bi-encoder alone. Both models are 0.6B parameters.
-
WhenYou are building or evaluating an LLM agent on tasks that span many sequential steps where early action choices constrain later options and mistakes compound across the trajectory.
UseAdd a bounded lookahead layer before each action selection. At each decision point, simulate N candidate action trajectories to depth D using the LLM itself, score the resulting states against an estimated future-reward signal (a lightweight value function or LLM-scored desirability), and apply a UCB-style exploration term to balance high-reward candidates against less-explored ones. Select the action from the highest-scoring branch. Keep D small (3 to 5 steps) so lookahead overhead stays bounded. Measure recovery rate (fraction of tasks where the agent recovers after an early mistake) and first-error step before and after enabling lookahead to validate the intervention.
EvidenceFLARE (Future-aware Lookahead with Reward Estimation), which adds UCB-style tree search and backward value propagation to standard step-by-step LLM reasoning, raised recovery rates from 5.4% to 29.7% on a long-horizon planning benchmark, and pushed the mean first-error step from 1.6 to 3.2. LLaMA-8B with FLARE outperformed GPT-4o running standard step-by-step reasoning on the same tasks. The gains trace to lookahead, not model scale.
-
WhenYour agent shows reasonable success on short tasks but degrades sharply as task length grows, or post-task analysis reveals early missteps that snowballed into states the agent could not recover from.
UseReplace single-shot full-sequence planning with a receding-horizon pattern: at each decision point, commit only to the next K steps, execute them, then re-plan from the freshly observed state. Do not try to recover a plan generated from imperfect early-task information. Set K empirically by measuring where first-error step occurs in failed traces; commit shorter than that threshold. Log the myopic-trap rate (tasks where the agent made an irreversible early commitment that determined the outcome by step 3) alongside standard success rate to confirm the intervention is hitting the right failure mode.
EvidenceOn a long-horizon planning benchmark, standard step-by-step reasoning showed a myopic-trap rate of 55.6% and a mean first-error step of 1.6. FLARE’s limited-commitment component (receding-horizon replanning) reduced the myopic-trap rate to 17.8% and pushed first-error step to 3.2. These gains were attributable to the replanning structure rather than model capability or prompting, as LLaMA-8B with the intervention outperformed GPT-4o without it.
-
WhenYou are building or refactoring the memory layer of a personalized conversational agent and deciding how to expose user history to the model.
UseExpose memory at 2 to 4 distinct granularity levels as separate callable tools rather than a single flat retrieval endpoint. At minimum, split into a high-level summary layer (user profile or topic track) and a detail layer (typed records or raw transcripts). Give each tool structured, typed output so the agent can route by category rather than parsing unstructured context. Measure answer quality and tool call count separately before and after the split to establish that the agent uses the layers selectively.
EvidenceNapMem (Xu, Sun et al. 2026, arXiv:2607.05794) organized user history into a four-layer pyramid: raw conversations, typed memory records, topic tracks, and user profiles. Each layer was exposed as a callable tool with provenance links to the layer below it. Ablations showed that adding memory granularity (the pyramid structure) contributed independently to accuracy gains beyond what active navigation alone produced. The structured layering enabled selective access patterns that single-endpoint retrieval cannot produce.
-
WhenYou are running multiple coding or research agents in parallel on an open-ended optimization task (algorithm improvement, architecture search, hypothesis generation) and find them converging on similar approaches within a few iterations, reducing diversity and missing potentially better solutions.
UseAssign each agent to a separate isolated execution context (a git branch, ephemeral container, or independent session). Keep a global coordinator with visibility across agents, but prevent agents from reading each other’s working state mid-exploration. Have the coordinator steer high-level direction (suggest regions to explore, flag promising or exhausted approaches) without exposing in-progress work from other agents. After each round, compare best-of-population quality against a single-agent baseline before deciding whether to scale fleet size.
EvidenceOn 13 of 15 open-ended optimization tasks, a two-tier system (Shepherd Agent with global context + isolated Search Agents each in a separate git branch) matched or outperformed SOTA LLM-guided evolution methods. The isolation mechanism was identified as the primary driver: agents forced to develop independent approaches produced a more diverse population than agents that could observe each other’s working state, yielding a higher best-of-population ceiling. (Virk, Edds, Xia & Zhang, UIUC, 2026.)
-
WhenYou are designing a multi-agent system where agents share a working memory, message bus, or common context window, and you suspect information sharing is causing agents to align on the same approach before each has independently explored its direction.
UseTreat context isolation as a first-class architectural control, not merely a resource constraint. Use separate branches, sandboxes, or session contexts as the primary mechanism for enforcing exploration diversity. Permit agents to share results only after committing to a direction, not while mid-exploration. If the system allows N agents, run a comparison of N isolated agents versus N context-sharing agents, measuring variance of approaches and best-of-population quality before collapsing to a shared context by default.
EvidenceThe SwarmResearch architecture deliberately used git-branch isolation as its diversity mechanism: each Search Agent began from the same base state but could not observe what other agents were writing. The structural isolation, rather than prompt-level diversity instructions, prevented early convergence. The 13/15 open-ended task result was attributed to keeping the population in a genuine exploration regime rather than an exploitation regime triggered by observing a successful neighbor. (Virk, Edds, Xia & Zhang, UIUC, 2026.)
-
WhenYou are running a coding or retrieval agent on a production workload that mixes simple bounded tasks (single-file edits, targeted lookups, constrained rewrites) with more open-ended tasks, and you pay per token or per API call.
UseAdd a lightweight difficulty classification call at the front of your agent loop, before the agent reads any files or invokes any tools. The classifier takes only the task description as input and returns a difficulty tier (simple, medium, complex). For tasks rated simple, cap the number of files the agent may read and limit context size to the minimum local neighborhood of the target. Execute the minimum-scope path. Run a verification step (unit tests, schema check, or model self-evaluation) on the result. Expand scope and re-run only if verification fails. Track cost per solved task before and after deploying the classifier, and compare the expand-trigger fire rate against your expected rate of misclassified-simple tasks.
EvidenceYin & Feng (2026) measured Agent Cognitive Redundancy on MSE-Bench: agents routinely applied 5-10x more compute than simple 121-task deterministic edits required, wasting 80-90% of their budget on work the task did not need. Their E3 framework (Estimate, Execute, Expand) placed difficulty classification before execution and matched the strongest baseline’s 100% task success while cutting cost 85%, tokens 91%, and files read 92% on the same benchmark. Results were corroborated on a live gpt-4o harness. Distinct from budget-prediction (BAGEN) and early-termination approaches (LaTER, BoundaryRouter): those route or stop based on mid-task signals; E3 right-sizes the initial read scope against a pre-execution difficulty estimate.
-
WhenYou are building or auditing a multi-model agent pipeline where different candidate models or tools handle different reasoning steps, and your current routing is based on static function labels, similarity scores against capability descriptions, or a cascade that tries the cheapest model first and escalates on failure.
UseBefore changing models or adding capacity, measure whether your routing actually sends each step to the model that performs best on it. Sample 50 to 100 recent reasoning chains and, for each step, run the same sub-task against at least two candidate models you have available. Record which model your router chose and which model produced the more accurate result. If the router’s choices and the best-performer diverge in more than 20% of steps, treat routing quality as the binding constraint before optimizing model selection or adding more compute. If building a new routing layer: require each candidate model to express calibrated self-confidence per step type (using historical accuracy as the calibration signal, not raw model confidence), award each step to the highest calibrated bidder, and measure accuracy per step type before and after.
EvidenceAgora (Zhou, Leonardis & Feng, University of Birmingham, 2026) replaced static agent routing with an incentive-compatible auction: candidate models bid per reasoning step based on rectified self-confidence, and the mechanism makes honest bidding the dominant strategy. On MMLU-Pro, Agora reached 71.9% versus 68.1% for the best single-model baseline in the same candidate pool. On MuSiQue-Ans multi-hop QA, it reached 43.0 EM / 54.3 F1. Improvements held across all five benchmarks tested against single-model, routing, and cascade baselines. The paper attributes the gain to the rectification step: raw self-confidence estimates are unreliable routing signals because models are systematically overconfident on task types where they underperform.
-
WhenYour long-horizon agent uses a fixed-size context window and relies on a static summarizer or truncation to handle context overflow mid-task, and you are fine-tuning the agent with reinforcement learning.
UseInclude context compaction as a trained RL behavior rather than leaving it as a separate inference-time heuristic. Extend your advantage estimation so credit propagates across compaction boundaries: actions before a summary event should receive reward signal from outcomes that occur after the compressed context continues. Apply token-level loss normalization so the summary-generation training signal is not crowded out by the task-execution signal. Before training, instrument the agent to record which information from pre-compaction context is referenced in post-compaction steps; this gap measurement tells you whether your current summarizer is losing task-critical information. Compare task completion rate under a fixed peak context budget, not raw token count, as the primary metric.
EvidenceCompactionRL (Li, Hou, Jing, Tang & Dong, Tsinghua University, 2026) jointly trained task execution and summary generation inside RL on GLM-4.5-Air. Token-level loss normalization and cross-trajectory GAE (Generalized Advantage Estimation) enabled credit to flow across compaction events. Under a fixed peak context budget, the approach raised SWE-bench Verified pass@1 from 59.8% to 66.8% (+7.0pp) and Terminal-Bench 2.0 from 21.4% to 24.5% (+3.1pp). Ablations confirmed that both mechanisms contributed independently: removing either component degraded results.
-
WhenYou are setting cost-reduction targets for an API-based agent and have not yet profiled where your spend is concentrated across token types.
UseProfile the cached-to-uncached input token ratio in a representative sample of your production sessions before choosing which cost driver to target. If cached input tokens account for more than 60% of your reconstructed cost, prioritize interventions that reduce cache invalidation, improve system-prompt prefix stability, or restructure tool schemas for maximum reuse over interventions that compress output volume.
EvidenceAcross the PointFive study’s measured workloads, cached input tokens accounted for roughly 87% of four-component cost and 80% of the actual billed amount, because the same instructions, tool definitions, and system context are transmitted on every agent turn at sharply discounted cache rates. Output-side compression addresses the remaining minority of spend. Without profiling, teams risk optimizing the small bucket while leaving the dominant cost driver unchanged. (Weinberger & Hozez 2026.)
-
WhenYou are building a retrieval layer for a long-context agent or RAG system that must handle corpora of hundreds of thousands to millions of tokens and are defaulting to large (7B+) model-based retrievers because sub-billion models seemed under-powered.
UseBenchmark a purpose-built sub-1B retriever (BlockSearch architecture or equivalent) against your current approach on your target corpus size before committing to a large retriever. Evaluate on your actual million-token workload rather than on short-context benchmarks, because the performance gap reverses at scale: small retrievers designed for this regime outperform large general-purpose models. Use the small retriever as the first stage and reserve a larger model for reranking the narrow candidate set it returns.
EvidenceBlockSearch (0.6B parameters) outperformed a 4B-parameter competitor by a substantial margin on MS MARCO and Natural Questions at million-token scale, and generalized to 10x the training context length without fine-tuning. The advantage arose because BlockSearch’s architecture was optimized for the softmax-dilution regime rather than adapted from a shorter-context general-purpose backbone. The result challenges the assumption that retrieval quality scales with model size when operating at extreme context lengths.
-
WhenYour LLM agent selects tools via a retriever with a fixed-k or similarity-threshold cutoff, and the candidate tools have meaningfully different per-call costs (token count, latency, permission scope, or blast radius).
UseReplace the score-only cutoff with a cost-aware stopping rule that walks the ranked tool prefix and stops when the marginal payoff of the next tool falls below its marginal cost. Start with a compact interpretable variant (score gradient, cumulative cost, and prefix position as features) so the rule is auditable at the permission boundary. Report loaded-tools-per-query and tokens-per-query alongside task success on internal evals.
EvidenceFeng, Zhang, Cheng and Qi (2026) prove that no rule consuming only the retriever’s scores can be Bayes-optimal once tool costs are heterogeneous, no matter how well the threshold is tuned. Their CAM-DF stopping rule, trained on the offline gap between stopping now and the best continuation, cuts tools per query by 37% at matched task success across 1,343 tasks in five tool-use domains, and beats a tuned predict-then-threshold baseline in all 20 tested permutations at cost heterogeneity d = 1.0. CAM-DF is a training-free plugin over any retriever; no LLM fine-tuning required.
-
WhenYou are building an agent that writes, indexes, or retrieves memory across turns and your per-turn LLM token spend on memory operations is a material cost or latency concern.
UseReplace LLM-driven memory writes and consolidation steps with a deterministic trace store. Organize raw interaction traces into two structures: an entity-context graph for cross-turn relational lookups and a temporal hierarchy for within-conversation recency weighting. At query time, combine evidence from both views using a scoring function calibrated on your own trace distribution, discarding conflicting entries before passing the result to the final-answer model. Reserve LLM calls for the final question-answering step only. Instrument per-turn token spend on memory operations as a baseline before switching, and verify answer quality on a held-out set after.
EvidenceZero-Mem eliminated LLM tokens from every memory operation except final QA by organizing raw traces as an entity-context graph plus a temporal hierarchy and combining them deterministically per query. Measured against the fastest LLM-based memory baseline at matched answer quality, Zero-Mem reduced memory-operation time cost by 57.6% while consuming zero LLM tokens on memory writes, indexing, and retrieval steps.
-
WhenYou are designing or tuning a hierarchical multi-agent search system with separated roles for query decomposition and retrieval execution, and you are deciding where to allocate model-tier budget across those roles.
UsePut the highest-capacity model on the delegation layer responsible for decomposing the query into sub-questions and directing the search. Keep execution sub-agents at a smaller model tier. Measure the accuracy delta from scaling each role independently, holding the other fixed, before assigning budget. Report exact-match accuracy and token spend per role separately to see which layer is the binding constraint. Do not assume the bottleneck is at execution without running the delegation sweep first.
EvidenceControlled capacity sweeps across a three-role hierarchical search system (delegation, execution, fixed answer writer) on five multi-hop QA benchmarks show that scaling the delegation backbone lifts exact-match accuracy by approximately 11 points while scaling execution sub-agents lifts it by approximately 2.6 points. The asymmetry holds across benchmarks, identifying decomposition quality as the primary accuracy lever in this architecture class.
-
WhenYou have a context-pruning step in a multi-stage research agent pipeline and want to reduce end-to-end token spend without retraining or replacing the pruning model.
UseAudit where your pruning step currently sits (pre-retrieval, post-retrieval, or pre-synthesis). If it is post-retrieval or pre-synthesis, move it to pre-retrieval: filter candidate sources by a relevance signal before the retrieval call is issued. Run both the original and relocated configurations on a fixed eval set and measure end-to-end token spend at matched answer quality. Make this configuration change before evaluating whether a better scoring model is needed.
EvidenceKolukuluru, Dernoncourt, Rossi, Lipka et al. (2026) ran the first systematic stage-aware comparison across three pipeline positions (pre-retrieval, post-retrieval, pre-synthesis) and two scoring approaches (lightweight heuristics, learned marginal-value model) in long-horizon research agents. Pre-retrieval pruning cut end-to-end token usage by up to 73% with little quality degradation. The position effect exceeded the scoring-method effect in all tested conditions. Savings compound because filtering before retrieval reduces what gets retrieved, accumulated, and passed to synthesis.
-
WhenYou are deciding between self-training (using the model’s own outputs as training data) and external distillation (using a stronger teacher model’s outputs) to improve performance on hard reasoning problems the base model rarely solves.
UsePrefer external distillation when the goal is expanding coverage of problems the base model fails on, not sharpening problems it already handles. Use self-training only where a stronger teacher is unavailable and accept that, under corrected measurement, capability gain evidence is absent and there is a measurable risk of corrupting problems the base model was already solving. Track per-problem corruption rate (problems correct before training, incorrect after) alongside aggregate accuracy for any self-training run.
EvidenceXu, Yan, Chen & Kechadi (2026) found that external distillation improves problems the base model rarely reaches while three forms of self-training do not, and that regression rejects the possibility that this difference is a byproduct of distillation’s larger overall gain (p < 1e-8). Self-training also corrupts problems solved at baseline at rates above the measured noise floor, a cost hidden by single-decode evaluation.
-
WhenYou are selecting between frontier-tier models for a production agent deployment and want to compare tool-selection safety, not just task accuracy.
UseRun at least one capability-mirage canary per candidate model before making a model selection decision. A capability mirage is a tool whose description fits the task but whose actual execution requires a permission, scope, or prior context the agent does not have. Do not infer selection safety from benchmark score or model scale: within a provider family, a cheaper model can have lower CSR than the flagship. Select based on directly measured susceptibility on your own tool domain, not on published leaderboard position.
EvidenceAcross the six canary types tested, capability mirages were the only type that reliably trapped frontier-tier models. The other five types became progressively easier for larger models to avoid. The most canary-susceptible hosted model in the study was mid-tier, not small. Within a single provider family, the lower-cost model had a lower CSR in at least one configuration. Capability tier alone was not a reliable predictor of selection safety.
-
WhenYou are deploying a long-horizon agent that generates multi-step action sequences and you want to reduce wasted compute on runs that are unlikely to succeed.
UseAdd a supervisor process that monitors the agent’s live action stream and aborts runs when intermediate progress signals fall below a calibrated threshold. Define per-task abort criteria before deployment: minimum steps completed before abort is allowed, the progress signal used (e.g. task sub-goal completion, tool-call success rate), and the threshold below which abort fires. After aborting, record the failure mode, route it to a distillation step, and store the resulting skill or memory entry in the agent’s in-session store before retrying or moving to the next task.
EvidencePILOT’s supervisor-worker harness on Terminal-Bench 2.0 reduced output tokens by 42.9% (GLM-5.1 backbone) and 47.4% (Kimi-K2.6 backbone) compared to running each task to completion regardless of trajectory. Because aborts free compute budget that redeploys to more promising runs, successful evaluations per million output tokens rose 110.3% and 134.0% respectively. The system ranked first in 5 of 6 benchmark configurations across the two backbone models.
-
WhenYou are running an agent on a long-horizon task and the agent fails partway through, and you want the remaining tasks in the same session or run to benefit from that failure.
UseAfter a failed run, extract the failure mode as a structured entry (what the agent attempted, at what step, and how it failed) and store it as either a skill (a generalized procedure for avoiding the same mistake) or a memory entry (a specific fact about the environment or task type that was wrong). Write the entry to the agent’s in-session store before the next run begins. Gate the distillation step on whether the failure is novel: compare against existing entries by embedding similarity before writing, and skip the write if a matching entry already exists.
EvidencePILOT’s Live Self-Evolution component, which performs this distillation within the same session, contributed 14.6 points of task success improvement on GLM-5.1 and 12.4 points on Kimi-K2.6 above the base model in the self-improvement configuration. These gains are separate from the compute-efficiency gains produced by the abort mechanism, indicating that in-session distillation adds value independent of the run-management component.
-
WhenYou are considering post-training a deployed LLM to use linear attention in order to reduce KV-cache memory and inference cost.
UseImplement SWA (Sliding Window Attention) with attention sinks first as a training-free baseline. Evaluate it on at least one long-context reasoning benchmark, Needle-in-a-Haystack being the lowest-effort starting point. If SWA meets your performance targets, skip the post-training run. Only proceed with linear attention post-training if SWA fails to meet your requirements and you have evidence that post-training will close the gap on long-context reasoning specifically, not just general benchmarks.
EvidenceJolicoeur-Martineau, Sukthanker, Cameron and Gervais (2026) found that SWA with attention sinks, which requires no post-training, matched post-trained linear attention on general downstream tasks and outperformed it by a factor of 2 to 10 on Needle-in-a-Haystack and BABILong long-context reasoning benchmarks. Prior linear-attention retrofit literature had not included this training-free control arm. The authors conclude that linear attention post-training likely needs to be done from scratch, or with substantially more extensive post-training than current methods use, to reach parity on long-context reasoning.
-
WhenYou are deciding whether to use a multi-call orchestration pattern (Self-Refine, Best-of-N, Debate, or similar) instead of a single-call approach, and you have not yet run both under equivalent optimization budgets on your target backbone and task domain.
UseBefore committing to a multi-call orchestration pattern, establish an optimized single-call baseline (chain-of-thought with equivalent prompt-optimization effort) and measure cost per unit of performance gain. If the gain is less than 5 percentage points at 2x or more token cost, default to the single-call approach unless your task domain has specific evidence favoring orchestration on your chosen backbone.
EvidenceLeins et al. (2026) compared Self-Refine, Best-of-N, and Debate against task-only and chain-of-thought baselines across 5 backbone models and 3 task domains (competitive programming, chess puzzles, mathematics), using GEPA to equalize prompt-optimization effort across all methods. Maximum observed gain from orchestration over optimized CoT was 4.6 pp; over task-only, 4.5 pp. Token cost was 2–4x higher. No statistically significant interaction between task difficulty and orchestration benefit was observed. Without equal-effort optimization, single-call baselines are systematically underrepresented, making orchestration appear more beneficial than it is.
-
WhenYou have selected an orchestration method (Self-Refine, Best-of-N, Debate, or similar) that performed well on one LLM backbone and are planning to apply it to a different backbone model.
UseRe-evaluate the orchestration method on the new backbone before deploying. Do not assume that a multi-call pattern that outperformed alternatives on one model will do so on another. Treat backbone as a primary configuration variable, not a plug-in swap.
EvidenceLeins et al. (2026) found strong method-by-backbone interaction effects in a controlled five-backbone study: the orchestration method that ranked first on one backbone frequently did not rank first on others. Task domain and difficulty level showed no consistent moderating effect on orchestration benefit, but backbone choice did. This means benchmark results for orchestration patterns are not portable across model families without re-validation.