The gains were real.
The measurement wasn't.
A frozen, untrained model run through the standard self-improvement evaluation pipeline produces results that look like genuine learning. Seven specific failures in how these pipelines are measured each flip a published finding when a proper null is present. The lesson isn't that self-improvement is impossible. It's that no one had checked whether the measurement worked.
First surfaced in Tandemly Briefing — 2026-08-20.
Measuring improvement
without a null is not measuring.
Self-improvement is one of the most active research directions in LLMs. The standard evaluation has a structural gap that none of the published work had filled: no one ran an untrained model through the same pipeline to see what it would score.
The premise of LLM self-improvement is appealing: a model generates its own training examples, trains on them, and repeats. No external labels needed. Several published papers report gains from this approach, measured by the standard method: run the model on a test set before and after training, then compare scores.
The problem is that this comparison is doing more work than researchers realized. Evaluation pipelines have configuration choices, and different configurations produce different scores on the same model. A single greedy decode per problem, with different GPU memory layouts or batch compositions across runs, shifts decoding behavior in ways that look like learning. Statistical tests applied to per-problem correctness bits without false-discovery-rate control reject the null too easily on data that is pure noise.
The field was reporting improvements against baselines that were themselves unreliable. No one had run a frozen, untrained model through the same pipeline to establish what the noise floor looked like.
A proper null for an LLM evaluation pipeline runs the unmodified base model through the identical procedure as the trained model, multiple times, and records the distribution of score changes. Any training effect must exceed that distribution to be real. This check was absent from the self-improvement literature the paper audits.
Run the untrained model.
Count what breaks.
Xu, Yan, Chen and Kechadi ran a frozen Qwen3-8B through the exact same evaluation pipeline as three rounds of rank-32 LoRA self-training. Then they cataloged every place where the pipeline produced a non-zero signal on a model that had learned nothing.
The design is straightforward. Take a base model. Do not train it. Run it through the evaluation pipeline used to report self-improvement results. If the pipeline is a reliable instrument, the frozen model should show no capability changes. What actually happened is that it did show changes, across seven measurable dimensions.
The first failure is single-decode variance. Most evaluations use one greedy decode per problem, run at a fixed temperature of zero. On the surface this seems deterministic. In practice, different batch compositions and GPU memory states change decoding outputs even at temperature zero, because floating-point operations in attention are not bitwise-stable across configurations. A ledger built on single-decode differences between two configurations treats this artifact as signal.
The second failure is the expansion statistic, a metric used in prior work to separate "acquisition" (solving problems the model never solved before) from "sharpening" (becoming more consistent on problems already in range). The researchers ran this statistic on the untrained frozen model and found it assigned an acquisition rate of 0.280. A model that was never trained appeared to have acquired new capabilities, because the statistic is sensitive to the same decoding variance that drives the first failure.
The researchers also tested a proposed fix called natural threshold repair, which calibrates the evaluation cutoff based on observed score distributions. It did not survive replication either. They replaced it with a per-problem exact test against a pooled baseline of multiple rollouts, with false-discovery-rate control applied across the full problem set. Under this corrected procedure, the trained models showed no statistically detectable improvement on any held-out replicate.
The null costs no new experiments to build. It reuses the baseline replicates a multi-arm study already runs. The frozen model is an additional arm in that pool, run through the identical procedure. The score distribution across its rollouts defines the noise floor. Code and evaluation artifacts are released alongside the paper.
Seven failures.
Zero verified improvements.
Under measurement that includes a proper null, three forms of self-training show no statistically detectable improvement on held-out replicates. External distillation does improve performance, but on a different axis than self-training claimed to.
The gap between distillation and self-training is the paper's most concrete substantive result. The two methods look similar from the outside: both update model weights on reasoning examples, both are evaluated on the same benchmarks. But under measurement with a proper null, they produce completely different outcomes. Distillation improves; self-training does not, and it degrades what was already working.
The authors note a practical consequence: teams using self-training because they cannot access a stronger teacher model are operating without verified evidence that the approach works. The measurement results do not support the same confidence level that the published self-improvement literature had implied.
The released code makes the null-construction procedure reproducible. It requires no new experiments. A study that already runs multiple baseline rollouts to estimate variance already has the raw material for a proper null. The contribution is the procedure for using those rollouts to set the noise floor, not a new data collection requirement.
The experiments focus on Qwen3-8B with rank-32 LoRA self-training evaluated on reasoning benchmarks. The measurement failures identified are likely to generalize across models and training configurations because they arise from properties of the evaluation pipeline rather than the specific model or method. But the authors' specific quantitative claims are bounded to the settings they tested.
What to do
with this.
For practitioners evaluating fine-tuning, self-improvement, or any method that claims capability gains on a reasoning benchmark. For anyone deciding whether to adopt self-training over distillation.
Where to begin.
Concrete steps for teams who want to apply the null-construction procedure to their existing evaluation work.
1. Identify your baseline rollouts. If your study runs the base model multiple times to estimate variance, you already have the raw material. The null model is one additional pass for the frozen, unmodified base, through the exact same evaluation pipeline you use for the trained models.
2. Download the released code. The paper's evaluation artifacts are available at the arXiv page (https://arxiv.org/abs/2608.20290). The null-construction procedure and per-problem FDR correction are implemented and documented there. Running them on your existing rollout data is the fastest path to a valid null.
3. Re-run your expansion statistic on the null model. If you have used an acquisition/sharpening split in any published or internal evaluation, run the same statistic on your frozen base model and check the rate. A non-zero result indicates the statistic is measuring noise, not capability transfer.
4. Add a per-problem corruption report alongside your accuracy delta. For any fine-tuning evaluation, add a column tracking how many problems that were correct in the base model became incorrect in the fine-tuned model. This is the cost side of the ledger that aggregate accuracy hides.
5. Read the paper. Xu, Yan, Chen & Kechadi (2026). Phantom Gains: Auditing Self-Improvement Against a Measured Null. arXiv:2608.20290. The full taxonomy of seven measurement failures, the replication data, and the corrected statistical procedures are there. The result that most shapes the practical recommendation: distillation improves hard problems, self-training does not (p < 1e-8).