The Benchmark Isn't
Measuring Context.
First surfaced in Tandemly Briefing — 2026-08-04.
When researchers want to test a model at a shorter context length, the standard approach is to drop content from the middle of the document. A new paper asks what that dropped content actually was. The answer, in many benchmarks: signal, not filler. The accuracy drop that follows may be measuring methodology, not window limits.
Measuring context-length
requires shortening the test.
The standard way to produce a shorter context is to delete from the middle. That turns out to be a problem when the middle contains the information the model was supposed to use.
Long-context benchmarks are designed to test how well a model handles large amounts of text. To study how performance changes as context grows, researchers run the same model at multiple prompt lengths: full length, three-quarters, half, one-quarter. The natural question is what to remove.
The dominant practice is to truncate from the middle. You keep the beginning and end, delete some content in between, and repeat at each length target. The assumption is that middle content is mostly filler, padding, or distractor material that is unlikely to contain the facts the model needs to answer the question.
That assumption is specific to how each benchmark was constructed. Many benchmarks embed task-relevant facts throughout the document, not just at the edges. When you delete from the middle, you remove signal along with noise. The model then fails not because it could not handle the context length, but because you gave it less of what it needed to answer correctly.
Published accuracy-vs-context-length curves built on naive truncation would reflect this confound. The downward slope you see as context shrinks could partly be measuring how much task-relevant content was removed, not how much the model struggled with length.
If we protect task-relevant content when shortening prompts and remove only the genuinely non-essential material, does the accuracy degradation curve change? And if it does, how much of the degradation we attributed to context-length limits was actually a truncation artifact?
Same benchmarks. Same models.
Different truncation.
Arjmandi ran two long-context benchmarks at four retention fractions under two truncation strategies, across four models from two providers. Replication followed on two additional benchmarks.
The study compared naive truncation, the standard middle-drop approach, against distractor-aware truncation, which first identifies which spans in the prompt are task-relevant and marks them for preservation. Only the remaining non-essential content is eligible for removal. The result is a shorter prompt at the same target length, but one where the signal has been protected.
The primary benchmarks were BABILong, a long-context reasoning benchmark, and GraphWalks BFS, which requires multi-hop traversal of relational information embedded in a long document. Both were run at four retention fractions: 100% of the original length (full context, used as baseline), 75%, 50%, and 25%. At each fraction, both truncation methods produced prompts of the same final length, so any accuracy difference reflects the truncation method rather than the amount of text the model received.
Model coverage included three Claude sizes, Haiku 4.5, Sonnet 4.6, and Opus 4.7, plus GPT-5.5 for cross-provider generality. Replication used MRCR v2 and Oolong, two additional benchmarks with different structural properties, to test whether the pattern held beyond the primary pair.
Because both truncation methods produce prompts at the same target length, the only variable is which content was removed. A difference in accuracy between the two conditions must come from the content that was present or absent, not from prompt length itself. That clean separation is what makes the study interpretable.
Naive truncation fell.
Distractor-aware held.
The two truncation strategies produced two distinct accuracy curves. The gap between them is an estimate of how much published long-context degradation is methodology rather than model.
The difference between the two curves at any retention fraction is an estimate of how much of the reported accuracy drop was caused by removing task-relevant content rather than by the context window being too short. It is not yet known what fraction of existing long-context degradation literature is affected by this confound, or by how much. What this paper establishes is that the confound exists, is measurable, and is large enough to produce statistically significant effects on well-known benchmarks with widely-used models.
Replication on MRCR v2 and Oolong confirmed the core pattern held beyond BABILong and GraphWalks BFS, which reduces the chance that the finding is benchmark-specific.
This paper tests four models across six benchmarks (two primary, four replication) at four retention fractions. That is a useful scope but not exhaustive. The finding does not prove that all long-context degradation is a measurement artifact, only that a portion of it can be. The magnitude of the confound likely varies by benchmark construction, task type, and how the task-relevant spans are distributed across the original document. Treating the paper as a warning flag for evaluation design is the appropriate response. Treating it as a full accounting of where long-context results stand is not.
What this changes
for evaluation practice.
The implication is not that context length is a solved problem. It is that some of the evidence used to size and justify long-context engineering work was measured with a flawed instrument. That changes the urgency on some items and the confidence level on others.
How to apply this
in your own evals.
The method requires no new models, no fine-tuning, and no special infrastructure. The bottleneck is span labeling, and that can often be done with a small amount of targeted annotation or rule-based markup.