Your retriever works fine.
Until the questions change.
A tool-retrieval system trained on one style of query covers 86.1% of requests from that source. Mixed into a production stream with other query styles, that number falls to 22.3%. The tool corpus hasn't changed. The correct answers haven't changed. The only thing that changed is how the questions are phrased.
First surfaced in Tandemly Briefing — 2026-08-17.
Same tools.
Different questions. Total failure.
Tool-retrieval benchmarks typically test a system on the same kind of queries it was trained on. Production doesn't work that way. And the gap between these two conditions is larger than almost anyone has measured.
When you deploy a tool-retrieval system, queries arrive from many places at once. A developer writes system-prompt instructions in one style. A user types a free-text request in another. An agent generates a sub-call in a third. An API integration formats its requests in a fourth. Each source writes queries differently, and each one phrased the same underlying need in its own vocabulary and structure.
The problem is that a retriever fine-tuned on queries from one of those sources doesn't automatically generalize to the others. It learned a pattern specific to its training distribution. When queries from a different source arrive, the pattern doesn't match, and the retriever fails. The tool corpus is unchanged. The correct tool for each request is the same as it always was. The failure is entirely in the mismatch between how the retriever learned to read questions and how the new queries are asking them.
This failure is invisible to most standard evaluations. Most benchmarks evaluate a retriever on the same source style it was trained on, or they mix sources without tracking which queries came from which source. The result is a reported coverage number that looks reasonable in testing and a system that quietly stops working in production whenever a new query source enters the mix.
When a tool retriever that works well on its training source is exposed to queries from a different source, what exactly fails, how bad is it, and what is the cheapest intervention that restores coverage without touching the tool corpus or retraining the retriever from scratch?
4,996 mixed queries.
One routing layer. 20 examples.
The researchers split a tool-query benchmark by source style, tested cross-source retrieval failure, ruled out formatting as the cause, then built a detection and routing intervention with a minimal labeled budget.
The study used ToolRet, a benchmark that spans multiple distinct source styles of tool-query pairs. The experimental design was clean: fine-tune a retriever on one source slice, then test it on other source slices of the same benchmark, keeping the tool corpus fixed throughout. This isolates the effect of query-source style from everything else that might vary between training and test conditions.
The researchers considered whether the failure might be a formatting artifact rather than a true style problem. Tool specifications can be presented in many formats: raw API schemas, executable function signatures, or structured skill cards with descriptions and examples. If the failure were simply about schema formatting, changing the tool representation would fix it. So they rerendered the tools as executable skill cards and ran the same cross-source test. The failure persisted. Source-style collapse is not a formatting problem.
The proposed fix, ToolScout, has two stages. In the first, each incoming query is fingerprinted using TF-IDF features, a bag-of-words frequency signature that doesn't require any embedding model or large dataset to compute. The signature is compared against seed examples from each known source style to determine which style the query most resembles. In the second stage, the query is routed to a retriever adapted for that style. The seed set required to make this work is 20 examples per source style, a number deliberately kept small to make the intervention practical without extensive labeling.
The full evaluation ran 4,996 queries across a mixed stream containing multiple source styles, measuring coverage and a coverage-weighted global top-1 proxy before and after applying the ToolScout routing layer.
The researchers compared TF-IDF fingerprints against semantic embedding similarities and length-based proxies as source-style detectors. TF-IDF outperformed the others. This is a mild but meaningful result: the signal that identifies which source a query came from is in its vocabulary and term frequencies, not in the geometric distance between query embeddings. You don't need a large embedding model to classify the style of an incoming query.
86 percent becomes 22.
Then back to 86.
Coverage collapse is sharp and the recovery is nearly complete. Both numbers matter for how you think about tool-retrieval systems in production.
A retriever achieving 86.1% coverage on queries from its training source falls to 22.3% when the same queries are mixed with queries from other sources. That is not a gradual degradation. It is a floor collapse. More than three-quarters of requests in a mixed production stream are uncovered, and standard aggregate task-success metrics don't surface this because they don't track which queries succeeded by source type.
The global top-1 proxy metric, which weights coverage across source styles, gives an even starker picture: 1.3% before ToolScout across five collapsed sources. A system at 1.3% global top-1 is effectively guessing at scale.
ToolScout raised coverage from 22.3% to 86.1% on the 4,996-query mixed stream, and the coverage-weighted global top-1 proxy from 1.3% to 53.9% across the five collapsed sources. The cost is 20 labeled examples per source style, TF-IDF computation, and no LLM fine-tuning. The retriever itself is not retrained. The tool corpus is not modified. The routing layer sits above both and redirects each query to the retriever most likely to handle its style.
The study uses ToolRet specifically, and the 20-examples-per-source threshold was sufficient for the five collapsed sources tested. Whether that threshold scales to more diverse or more numerous source styles in a real deployment is not established by this paper. The finding that TF-IDF outperforms semantic similarity for source-style detection is also measured within the benchmark's own query distribution; external replication would strengthen it.
The paper also rules out raw API-schema formatting as the sole cause of collapse but does not test every possible tool-representation format. The persistence of the failure under executable skill-card rendering is evidence against a pure formatting explanation, not proof that all formatting choices are irrelevant.
Fingerprint your queries
before tuning your retriever.
The failure is in the training distribution, not the tool content. That means the diagnostic comes before the fix, and the diagnostic is cheap.
Where to go
from here.
Source-style collapse is diagnosable and fixable with minimal investment. Here is a concrete path to applying what this paper found.