Not every token needs
the same compute.
Cornell researchers built a tiny policy network that rides alongside a frozen language model and decides, at each decode step, how much work the model actually needs to do. The result is a better quality-efficiency tradeoff than any fixed efficiency setting produces, with no retraining of the base model required.
First surfaced in Tandemly Briefing — 2026-05-26.
Fixed settings waste compute
on easy tokens.
Language model serving treats every decode step identically. There is no mechanism to distinguish a step that just needs to generate "the" from a step that needs to resolve an ambiguous reference. Both get the same compute budget.
When a language model generates a response, it works one token at a time. At each step, it runs the input through a stack of transformer layers: attention heads that look at context, feed-forward networks that process the attended information, and normalization steps throughout. Every layer runs for every token. The compute cost is constant per token, regardless of what the token is or how much reasoning the step demands.
Efficiency techniques have tried to address this by applying reductions uniformly: prune a fraction of attention heads, quantize activations to a lower bit-width, skip some feed-forward neurons. These work, but they apply the same reduction everywhere. You pick a configuration that trades some quality for speed, and it holds across every token in every generation, from the first word to the last.
The assumption baked into this approach is that all tokens are roughly equal in difficulty. They are not. Generating a word in a predictable context is very different from generating the next word in a complex clause with multiple possible continuations. The compute cost should, in principle, vary with the difficulty of the step. But existing serving stacks have no mechanism for that kind of adaptation.
Can a small, trainable policy network learn to allocate compute across the three main efficiency levers on a per-token basis, matching or beating the quality of the best fixed setting at the same average compute budget?
A policy that reads
the model's own signals.
SOL (Self Optimizing Language Models) pairs an unchanged, frozen base model with a lightweight policy network. The policy watches the model's hidden state and makes efficiency decisions at each decode step. The base model is never modified.
The design begins with a constraint: the base language model is frozen. No weights change. SOL's contribution is a separate, small policy network that runs alongside the base model at inference time. At each decode step, this policy network reads the model's intermediate hidden state, which is a high-dimensional vector that encodes what the model "knows" about the current context and where it is in the generation process.
From that hidden state, the policy selects one configuration from a menu of three efficiency knobs. Attention sparsity controls which attention head computations to skip for this step. Structured MLP pruning controls which feed-forward neurons to bypass. Activation quantization bit-width controls how precisely to represent intermediate values in the computation. These three categories cover the main sources of compute cost in a transformer layer.
The key is that the policy selects these settings independently at each token, not once for the whole generation. The policy is trained to learn which combinations of efficiency settings are safe for which types of hidden states. Easy, predictable steps get aggressive reductions. Complex, novel steps get closer to full compute. Because only the policy network is trained and the base model is frozen, the approach can potentially be applied to any model that exposes its hidden states at inference time.
| Efficiency action | What it controls | Compute lever |
|---|---|---|
| Attention sparsity | Which attention heads compute this step | Attention cost |
| Structured MLP pruning | Which feed-forward neurons activate this step | FFN cost |
| Quantization bit-width | Precision of intermediate activation values | Memory bandwidth and arithmetic cost |
Techniques like LaTER, BoundaryRouter, and Dual-Dimensional Consistency all operate at the level of reasoning trajectories or query routing: they decide whether to use a full agent, how many steps to take, or when to terminate self-consistency. SOL operates inside the decode loop at the per-token level. The two categories are orthogonal.
7.3 points on MMLU
at matched compute.
The core result: per-token compute selection beats uniform budget allocation on the quality-efficiency Pareto front. The gap is not at one operating point but across the full tradeoff curve.
SOL produces up to 7.3-point improvements in MMLU accuracy compared to uniform-budget allocation at the same total FLOPs. Uniform allocation applies the same efficiency reduction to every token in a generation. SOL's per-token decisions appear to concentrate savings on steps where quality loss is minimal and preserve compute for steps where it matters.
This synthesis is based on the editorial summary from the briefing that surfaced this paper; the full arXiv page was inaccessible during synthesis. The 7.3-point MMLU gain and Pareto front comparison are the specific numbers from that summary. Additional benchmarks, model families tested, and ablation studies may provide more detail than is captured here. The paper is at arxiv.org/abs/2605.10875.
What this means
for serving teams.
Per-token compute routing is a distinct optimization lever, separate from model choice, quantization level, and reasoning trajectory control. The frozen-base design makes it practical without requiring the ability to retrain the LLM.
Where to go
from here.
Starting points for teams that want to understand or apply per-token compute routing.