The missing baseline
that changed the math.
First surfaced in Tandemly Briefing — 2026-08-28.
The case for post-training LLMs to use Linear Attention rested on comparisons that left out a simpler alternative. Jolicoeur-Martineau, Sukthanker, Cameron and Gervais ran that comparison. Sliding Window Attention with attention sinks needs no post-training at all, matches or beats post-trained linear attention on general tasks, and outperforms it by 2 to 10 times on the long-context reasoning benchmarks that motivated the retrofit.
A published fix with
a missing baseline.
Standard transformer attention gets expensive as inputs grow long. The field's answer has been to retrofit deployed models with Linear Attention. That answer was never compared against the right control.
The key-value (KV) cache in a transformer grows with every new token. For a model processing long documents or extended conversation histories, that growth is quadratic: twice the context length means four times the memory and compute. At the scale modern serving infrastructure operates at, this is a real cost driver.
The field developed a response: post-train a deployed LLM to use Linear Attention. Linear attention replaces the quadratic operation with a matrix approximation that scales linearly. Because post-training modifies only the attention mechanism rather than retraining from scratch, it was presented as an efficient path to lower serving costs: one-time fine-tuning expense, ongoing savings at inference.
The problem is that this line of work developed without a proper control arm. Sliding Window Attention with attention sinks is a training-free alternative that limits how far back a token's attention reaches and reserves a few anchor tokens to absorb excess attention mass. It has no post-training requirement. But for a body of work arguing that post-training is worth the cost, no one ran SWA with attention sinks in the same evaluation.
If you add SWA with attention sinks to the comparison that motivated linear attention post-training, does post-training still look like the right call? The answer shapes how teams should think about the engineering budget for a linear-attention retrofit on any deployed model.
Adding the control arm
that was missing.
The method is conceptually clean: run SWA with attention sinks as a baseline in the same experiments used to evaluate post-trained linear attention. No post-training required for the baseline.
Sliding Window Attention confines each token's attention to a local window of recent context. Rather than attending to every prior token, the model sees only the most recent fixed number of tokens. This removes the quadratic memory growth at the cost of losing access to tokens outside the window.
The attention sinks piece addresses a known empirical quirk: early tokens in a sequence attract disproportionate attention mass even when they carry little relevant information. Without a designated place for that mass to go, nearby tokens get crowded out by it. Attention sinks designate the first few tokens as always-attended, so excess mass has a stable destination. No weights change. The modification is applied at inference time with no post-training step.
The evaluation ran across multiple LLMs and general downstream tasks, then extended to two benchmarks designed specifically to stress long-context reasoning. Needle-in-a-Haystack plants a specific fact at a known position in a long document and asks the model to retrieve it. BABILong requires multi-hop reasoning across a very long text. Both are commonly used to motivate and evaluate long-context improvements.
The post-trained linear attention models in the comparison had been fine-tuned on the same model families, which makes the trade-off legible: post-training compute plus inference savings on one side, inference savings alone on the other.
SWA matches on general tasks,
dominates on long-context.
On standard benchmarks, post-training bought nothing over a training-free baseline. On the benchmarks that motivated the retrofit, the gap ran in SWA's favor.
Across standard general downstream benchmarks, SWA with attention sinks matched post-trained linear attention. The post-training cost produced no gain over a training-free alternative on the evaluations the community most commonly runs to confirm quality is preserved during an efficiency intervention.
On Needle-in-a-Haystack and BABILong, SWA outperformed post-trained linear attention by a factor of 2 to 10. These are not marginal differences. They are the benchmarks that motivated linear-attention retrofits in the first place: if a model cannot retrieve a planted fact or reason across a long document, the long-context efficiency improvement has little practical value.
The authors draw an explicit conclusion from this gap. Linear attention post-training likely needs to be done from scratch, or with substantially more extensive post-training than current methods use, to reach parity with SWA on long-context reasoning. The retrofit approach, as it has been practiced, does not clear even the training-free bar on the tasks that matter most for long-context serving.
This comparison covers post-trained linear models only. From-scratch linear attention architectures, trained on large pretraining runs rather than retrofitted onto a deployed model, are a different design decision entirely and are outside the scope of this paper. The authors are explicit about this. The recommendation is about retrofitting; it does not settle whether linear attention makes sense as an architecture choice for a new pretraining run.
What changes if you
take this seriously.
The claim is that a commonly-budgeted engineering investment was never compared against the right baseline. That has direct implications for teams considering it now.
How to run the
right comparison.
Concrete steps for teams evaluating this.