Agent Evaluation · Tool Selection · LLM Safety

Task success hides
selection failures.

Researchers planted engineered decoy tools directly into agent tool sets to measure what aggregate task scores never show: whether the agent can distinguish plausible-but-wrong tools from the right one. Canary susceptibility varied by a factor of 36 across eight models tested. Capability tier predicted almost nothing. Frontier models had one specific type they could not reliably dodge.

Core concept
A canary tool is an engineered decoy placed in an agent's tool set. It probes whether the agent selects the wrong tool, separate from whether it can find the right one. Canary susceptibility rate (CSR) is the fraction of tasks where the agent picks at least one canary instead of proceeding correctly.

First surfaced in Tandemly Briefing — August 5, 2026.

scroll to explore
01The problem

Aggregate task score
is the wrong meter.

When agent tool sets grow, teams have two separate problems: retrieval (did the right tool make it into context?) and selection (did the agent actually use the right tool, not a plausible-but-wrong neighbor?). Only the second problem is being ignored.

Most agent evaluation ends with a pass or fail on a task. The task either got done or it did not. What that number does not capture is the path the agent took to get there. An agent that called two wrong tools and then stumbled onto the right one scores identically to an agent that went straight to the correct tool on the first try.

This gap matters practically. A task can succeed despite a bad tool-selection decision if the wrong tool returned a graceful error, or if a later correction happened to cancel out the earlier mistake. Teams using aggregate success rates as their signal have no idea how close to the edge they are operating. They cannot tell whether the agent is reasoning about tools or pattern-matching on surface similarity.

Prior work on tool retrieval addresses a related but different question: whether the right capability makes it into the context at all. If retrieval is the problem, you fix the retriever. Tool-selection failure is a separate failure mode that happens after retrieval, when the right tool is already present and the agent still reaches for the wrong one. That is what Anand and Chattaraj set out to measure.

The question this paper asks

If you place a plausible-but-wrong tool directly in front of an agent, next to the correct one, how often does it pick the wrong tool? And does the answer depend on which kind of wrong tool you use, or on which model is running the agent?

02The experiment

Six types of canary,
8,640 runs.

The team defined a six-type taxonomy of decoy tools, planted them at three density levels across 120 tasks, ran eight models at three random seeds, and used a provider-independent LLM judge cross-validated with a second judge to score every run.

A canary tool is not a random bad tool. Each type is engineered to probe a specific reasoning weakness. The team built six: a semantic decoy has a name and description similar to the correct tool but a different function; a parameter trap uses the right function signature but wrong argument types or constraints; a capability mirage describes a tool that looks applicable but requires permissions, scopes, or context the agent does not have; a prerequisite blindness canary assumes a prior step the agent has not taken; a temporal decoy is accurate for a different time window than the current task; and a granularity trap operates at the right intent but the wrong level of specificity.

Eight models spanning small open-weight, mid-tier hosted, and frontier-tier were tested. Each model ran 120 tasks at three canary-density conditions and three random seeds, producing 8,640 scored runs. An additional 2,880-run ablation re-ran the same tasks with softened canary phrasing stripped of any obvious give-away keywords. If CSR dropped with softening, the agent was avoiding canaries by word-matching rather than reasoning. If CSR stayed flat, the probe was measuring genuine reasoning.

A provider-independent LLM judge scored tool-selection decisions. A second judge corroborated the scores independently (Cohen's kappa = 0.75), confirming the measurement was not anchored to a single provider's bias.

Capability Mirage
Traps frontier models
Tool looks applicable but requires permissions, context, or scope the agent does not have. Frontier models fall for this type at rates close to smaller models. The only canary type that consistently fires on the largest models.
Semantic Decoy
Fires on smaller models
Name and description are similar to the correct tool; function is different. Tests whether the agent reads the whole description or anchors on label similarity.
Parameter Trap
Fires on smaller models
Right function signature, wrong argument types or constraints. Tests whether the agent checks argument compatibility or just matches on function intent.
Prerequisite Blindness
Fires on smaller models
Tool assumes a prior step the agent has not completed. Tests whether the agent tracks execution state across tool calls or treats each call as independent.
Temporal Decoy
Fires on smaller models
Accurate for a different time window than the current task. Tests whether the agent checks temporal validity or treats tool descriptions as always-current.
Granularity Trap
Fires on smaller models
Correct intent, wrong level of specificity. A bulk export tool when the task needs a single record, or vice versa. Tests resolution matching.
03Findings

36x spread, and
size predicts nothing.

Three findings that cut against standard practice: susceptibility varies more than an order of magnitude across models, capability tier is an unreliable proxy for selection safety, and frontier agents have a specific weak spot that does not go away with scale.

36x
Range of per-task canary
susceptibility rate across 8 models
−0.34
Spearman correlation: CSR predicts
downstream task failure
0.75
Cohen's kappa:
inter-judge agreement on CSR scoring
Finding 1: Capability tier is a poor proxy for selection safety

The model with the lowest CSR overall was Claude Opus 4.8. The highest was Llama 3.1 8B, which is a larger spread than expected from a strict capability-tier account. More importantly, the most canary-susceptible hosted model was mid-tier, not small. Within a single provider family, the cheaper model was sometimes the safer one on tool selection. Buying a bigger model does not reliably buy you better selection hygiene.

Finding 2: The taxonomy is capability-stratified

Capability mirages are the only canary type that reliably fires on frontier models. All six types fire on small open-weight models. As model scale increases, five of the six types become progressively easier to avoid. But capability mirages stayed dangerous across model sizes. The implication is that frontier-model tool-selection evals should weight this type heavily and treat the other five as sanity checks for small or distilled models.

Finding 3: Softening phrasing does not change frontier CSR

The 2,880-run ablation removed give-away phrasing from canary descriptions, checking whether agents were dodging canaries by word-matching rather than reasoning. For frontier models, CSR was essentially unchanged after softening. The probe is measuring actual reasoning about tool applicability, not surface phrase avoidance. That makes CSR a genuine behavioral metric, not an artifact of unusual wording.

Scope and limitations

Eight models, 120 tasks, three canary density conditions. The six canary types were defined by the authors; the taxonomy is not independently validated. CSR predicting downstream failure at rho = -0.34 is a moderate correlation, meaning susceptibility is a relevant signal but not a complete predictor of task outcome. The authors released the framework, canary schemas, tasks, and run logs, which allows third-party replication and extension.

04Practical takeaways

Add canary rate as
a standing metric.

The canary framework is designed to slot into a staging environment with no fine-tuning. The main move is treating CSR as a diagnostic instrument rather than inferring selection quality from aggregate task success.

1
For teams building production agent systems
Add CSR as a metric alongside task success in your staging environment. Task success rates tell you whether the agent gets things done; CSR tells you how close to a selection failure it ran on the way there. The two numbers can diverge significantly, and a high task success rate can coexist with a dangerously high CSR. Checking CSR first gives you early warning before a production tool set expands.
2
For teams selecting or comparing frontier models
Do not infer tool-selection safety from capability tier or benchmark score. Within a provider's model family, the cheaper model may have a lower CSR on tool selection. The correct approach is to run a canary probe directly on the models under consideration. Capability mirages are the type most relevant to frontier models; weight them accordingly when building your canary set.
3
For teams designing agent evals
Include a selection diagnostic as a first-class eval dimension. The existing agent-eval literature grades retrieval (does the right tool appear in context?) and task completion (does the task succeed?), but skips the middle step. A canary probe fills that gap without requiring a redesigned benchmark or new training data. The published canary schemas give you a starting point you can adapt to your own tool domain.
4
For teams tracking agent reliability over time
Run canary probes on a schedule, not just at initial deployment. CSR can drift as tool sets expand or model versions change. A model that passed an initial canary check may become more susceptible when your tool set grows from 20 to 100 tools. Scheduling periodic canary runs catches that drift before it shows up as task failures.
05Further exploration

Where to go
from here.

The framework is released and ready to use without fine-tuning.

1
Read the paper
Anand, A., & Chattaraj, S. (2026). Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools. arXiv:2608.04719. The paper includes the full canary taxonomy, scoring protocol, and run logs for replication.
2
Seed your staging tool set with capability mirages first
If you are using a frontier model, capability mirages are the highest-priority canary type. Write one canary per tool scope boundary in your system (one tool that describes an admin action the agent is not permitted to take, one that assumes a session context the agent does not have) and measure CSR across a sample of tasks before expanding your tool set.
3
Run the softening ablation as a validity check
After building your canary set, re-run with phrasing stripped of any obvious give-away signals. If CSR drops substantially, your canaries are testing word-matching, not reasoning. That tells you the canaries need harder phrasing before the metric is meaningful.
4
Pair with the related work on retrieval failure
This paper addresses selection, not retrieval. If your retriever is the bottleneck, see also the synthesis of Liu, James, Wang, Xiao & Lin (2026) on source-style collapse in tool retrieval: When Tool Retrieval Fails. The two diagnostics address different layers of the same tool-use stack.
5
Retrieve the framework, canary schemas, tasks, and run logs
The authors released all evaluation artifacts, so you can start with their existing task set and canary schemas and adapt them to your own tool domain before building entirely from scratch. Find them via the arXiv page at https://arxiv.org/abs/2608.04719.