Drowning
in Documents.
It's the math.

In-context retrieval is exactly what it sounds like: instead of indexing documents in a separate vector store and querying it, you put all the documents directly into the model's context window and let the model find what it needs by reading. It is conceptually clean, avoids the complexity of external retrieval pipelines, and has become more practical as context windows have expanded to hundreds of thousands or millions of tokens.

The problem is that it does not work reliably at scale. Measured accuracy on retrieval tasks falls as the number of documents in the context increases. Researchers and engineers who have observed this pattern have often attributed it to "context length limitations," as if the model simply runs out of useful memory past some threshold. This explanation is intuitive but vague. It does not tell you what is actually breaking, and it does not point toward a fix.

The more troubling observation is that the failure appears to happen even when the gold document is clearly within the model's context length and even when the model would correctly retrieve it in a shorter context. Something specific goes wrong as the document pool grows. The question is what, exactly, and whether it is fixable without changing the model.

The key insight starts with how attention works. When a transformer attends to a sequence, it computes a raw score for each position, then runs those scores through a softmax function. Softmax turns raw scores into a probability distribution that sums to one. The denominator of that distribution grows with every additional position in the context. More tokens means a larger denominator, which means each individual position receives a smaller slice of the total attention budget, regardless of its raw score.

This is attention dilution. A gold document might have the highest raw relevance score in the entire context, but if the context contains a thousand other documents, the softmax math distributes attention across all of them. The gold document's normalized attention weight collapses, not because the model assessed it as less relevant, but because the denominator got bigger. The model is being drowned out by arithmetic, not by confusion.

The researchers confirmed this with a controlled diagnostic: hold the gold document fixed, add distractor documents one at a time, and measure the model's normalized attention weight on the gold document at each step. The result is that normalized weight drops monotonically as distractor count increases, even when the gold document's raw pre-softmax score stays stable. This is the dilution signature, and it separates the failure class from genuine capacity limitations.

With the mechanism named, the team proposed two fixes. First, a length-aware adjusted attention softmax that compensates for denominator growth as the sequence lengthens. Second, document-level sparse attention, which directly limits which document positions attend to each other, preventing irrelevant documents from contributing to the denominator of attention over the gold document.

They also built BlockSearch, a 0.6 billion parameter language model trained specifically for block-level retrieval in long contexts. Rather than trying to patch the dilution problem inside a general-purpose large model, BlockSearch is designed from the ground up to work correctly as document counts scale into the millions. It length-generalizes up to ten times beyond its training context length, meaning a model trained on contexts of a given size can handle contexts ten times larger without retraining.