The harness
picks the winner.
One model. 26 equally defensible evaluation setups. A 58-percentage-point range in reported score. A new study finds that config-fragile benchmark items carry 95.7% of the score gap between adjacent models, and that four of twelve models can claim rank one depending entirely on how the harness is configured.
First surfaced in Tandemly Briefing — 2026-08-21.
A score is a band,
not a number.
What the paper did, and why it matters.
Researchers built a fragility grid: 12 open-weight instruction-tuned models from four families, each answering 3,679 items drawn from ARC, HellaSwag, MMLU, and TruthfulQA, under 26 equally defensible harness configurations. Items, weights, and decoding strategy stayed fixed. Only harness setup changed.
The result collapses the standard interpretation of leaderboard scores. One model (gemma4-31b) spans 31% to 89% accuracy on harness choice alone. Four of the twelve models can claim rank one under some configuration. And the items responsible for this variance are not random. Config-fragile items carry 95.7% of the score gap between adjacent model pairs on average. Benchmark compression, which is supposed to sharpen discrimination, makes the problem worse: it selects for the most fragile items, not the most stable ones.
The per-item records and analysis scripts were released with the paper. Every number reproduces on a CPU in seconds, so this is a claim anyone can verify.
The benchmark assumes
a neutral instrument.
Standard leaderboard practice treats score as capability. This paper tests that assumption.
LLM evaluation has converged on a small set of multiple-choice benchmarks: ARC, HellaSwag, MMLU, TruthfulQA. The Open LLM Leaderboard and similar rankings formalize the practice. A model gets a score. Teams use that score to choose between options. The implicit assumption is that the score measures something stable about the model.
Prior work showed aggregate variance under harness changes, but stopped there. A model's score shifted when you ran the same benchmark under a different harness setup, but which items were responsible? Were they distributed randomly, or concentrated in a small, identifiable subset? Did benchmark compression, the standard practice of distilling thousands of items into a smaller more discriminative set, help or hurt?
These questions stayed open. And they matter more than they might appear. If model-selection decisions rest on rankings that are partly manufactured by harness configuration rather than model capability, teams can be choosing systematically wrong models without any visible signal that something has gone wrong.
Which items drive the harness-induced variance, and can we identify them? If fragile items are concentrated and measurable, the problem has a fix. If they are diffuse and unpredictable, the problem is structural.
One grid to find
the fragile items.
The setup was deceptively simple. Every variable held fixed except one.
The paper builds what it calls a fragility grid. Twelve open-weight instruction-tuned models from four families answer 3,679 items from the four standard benchmarks under 26 harness configurations. Every configuration is equally defensible. None is wrong. Items, weights, and greedy decoding stay fixed across all configurations. Only the harness setup varies.
The harness configurations differed on choices like: how the question is formatted before the model sees it, how answer choices are presented, and most importantly, how the answer is scored. Should the harness record what text the model generates, or which answer option received the higher log-likelihood (the log of the model's probability for each choice)? Both methods appear in the wild. Neither is wrong. And the difference between them turns out to carry most of the variance.
The result is one correctness bit per model-item-config cell: a three-dimensional grid. An item is config-fragile if its correctness score changes across configs for a given model. This is the key diagnostic unit, and it turns out to be concentrated. Most items are stable across configs. A small fraction are not, and that fraction does almost all the work of separating adjacent models.
Four models can
claim rank one.
The numbers are large enough that none of them read as measurement noise.
Gemma4-31b scores between 31% and 89% depending solely on harness configuration. The model is the same. The benchmark items are the same. Decoding is greedy throughout. The 58-point range is entirely attributable to config choice. This is not an outlier: several other models in the study show comparable ranges. A single reported score on any of these benchmarks is a single draw from a wide distribution.
Of the twelve models in the study, four can achieve the top leaderboard ranking under some configuration. The harness is not a neutral instrument. It selects the winner. Teams relying on a published ranking to choose between models may be relying on a ranking whose top position changes with config choice rather than reflecting stable capability ordering.
On items where two adjacent models both answer consistently across configs, the pair scores essentially the same. The entire observed ranking difference between neighboring models comes from config-fragile items: the small fraction of benchmark questions whose correctness score changes with harness configuration. Config-fragile items carry 95.7% of the adjacent-model score gap on average. Remove those items, and adjacent models converge.
Benchmark compression is a standard practice: start with thousands of items, keep the most discriminative ones, end up with a smaller benchmark that still separates models. Item discrimination correlates with fragility at r = 0.28 (95% CI: 0.25-0.30). Selecting for discrimination is, on average, selecting for fragility. A compressed benchmark is more likely to be fragility-weighted than the full benchmark it was derived from. The discriminative property that compression maximizes and the config-fragile property that creates unstable rankings track together.
Of all the harness choices that vary across the 26 configurations, scoring mode drives the most variance. Whether the harness scores by what text the model generates, or by which answer option received the higher log-likelihood, is the choice that matters most. Option ordering, which is what most evaluation protocol standardization focuses on, turned out to matter much less. Evaluation protocols may be standardizing the wrong variable.
The study covers 12 models from 4 families, 3,679 items from 4 benchmarks, and 26 configurations. The findings describe these models and benchmarks under these configurations. Generalization to other model families, other benchmarks, or other configuration spaces requires further work. The paper's released code lets teams run the same analysis on their own setup, which is the appropriate way to check whether these findings apply to a specific use case.
What changes about
model selection.
The findings affect anyone who uses published leaderboard scores to choose or rank models.
Concrete next
steps.
What you can actually do with these findings.