When a language model scores lower on a freshly written copy of a benchmark than on the original, the drop has two possible causes. The model may have seen the original test items during training (contamination), or the new copy may simply differ a little in style or difficulty from the old one (natural distribution shift). Today these causes are confounded. The best-known replication study, GSM1k, admits it cannot fully separate them. We propose to separate them using a regularity from computer vision. Across many models, accuracy on a naturally shifted test set tends to fall on a straight line when plotted against accuracy on the original. A shift that affects every model the same way moves the whole line. Contamination affects only the models that saw the items, so it shows up as individual models falling below the line. We call that gap negative effective robustness, and we treat it, with an error bar derived from test-set size, as a contamination estimate.
This matters for two reasons. First, it turns every existing "fresh copy" of a benchmark into a per-model audit that needs no access to model weights or training data. Second, it gives an honest answer on benchmarks close to their ceiling. Near 100% accuracy, differences between models shrink below measurement noise, so the method will sometimes have to report that a residual is undecidable instead of producing a false accusation.
Core Claim
Measuring Robustness to Natural Distribution Shifts in Image Classification evaluated 204 ImageNet models. It found that accuracy on ImageNetV2, a natural re-collection of the ImageNet test set, is almost perfectly linear in original accuracy ($r^2 = 1.00$). It also found that robustness interventions almost never lift a model above that line. The line therefore behaves like a property of the dataset pair, not of any particular model.
Our claim is that the same structure holds for pairs of original and replicated language-model benchmarks. If it does, the part of a model's drop that the line cannot explain is a model-specific signal, and it lines up with exposure to the original items. Concretely, for model $m$ with original accuracy $p_m$ and replica accuracy $q_m$, we fit
$$\operatorname{logit}(q_m) = a + b\,\operatorname{logit}(p_m) + \varepsilon_m$$
over a reference set of models that are believed to be clean. The logit transform, $\log(p/(1-p))$, stretches scores near the ceiling, where raw accuracies bunch together. We then score a candidate model by $z_m = -\varepsilon_m / \sigma_m$, where $\sigma_m$ is the standard error of the residual. That standard error combines the binomial sampling noise of both test sets, propagated through the logit, with the uncertainty of the fitted line.
The noise term is not optional. When AI Benchmarks Plateau showed that test-set size is a primary driver of saturation, because leading models' score spread falls below the sampling error. The same sampling error bounds how small a contamination effect we can detect.
Relation to Prior Work
A Careful Examination of Large Language Model Performance on Grade School Arithmetic motivates this proposal and supplies its main testbed. The authors wrote GSM1k, 1,205 problems mirroring the GSM8k grade-school math benchmark. They found drops of up to 8% for some model families and negligible drops for frontier models. They also reported an $r^2 = 0.36$ correlation between a model's likelihood of generating GSM8k items and its drop. Their analysis uses the raw drop. That quantity mixes any GSM8k-to-GSM1k difficulty shift into every model's score, which is exactly the confound they acknowledge.
The ImageNet study above supplies the method. It used the line to show that interventions add no robustness. It never read points below the line as a diagnostic, and it never studied language models or contamination.
Dynamic and live benchmarks attack contamination by replacing static test sets. DyVal generates reasoning problems from graphs, and LiveBench rotates in post-cutoff tasks each month. Both constrain our design in a useful way. They show that fresh items expose contamination. LiveBench, however, cannot tell whether month-to-month changes reflect temporal drift, varying difficulty, or contamination. That is the same decomposition problem, and our line fit can be applied to consecutive LiveBench releases.
Leaderboard Incentives formalizes "benchmaxxing": fine-tuning on a benchmark's task rather than its exact items. On Winogrande, a model could overtake a competitor with 18 extra training steps. This raises a threat we must handle, because task-level overfitting may also push models below the line. We use their setup to build a control condition that tells the two apart.
Temporal and in-the-wild shift benchmarks such as Wild-Time and WILDS show that natural shift routinely costs 20% or more and that specialized methods rarely beat plain training. This constrains us too. It tells us to expect a large shared shift term that must be modeled, not ignored.
Research Program
The work has three parts.
Test whether the line exists. We first ask whether clean language models fall on a line for replicated benchmark pairs at all. The reference set is models whose pretraining corpora are public, so exact n-gram overlap with GSM8k can be checked. We add robust regression over all models, which assumes most models are near the line. That assumption is supported by the negligible frontier-model drops Zhang et al. report.
Validate against known contamination. We then plant contamination in a controlled way. Following the model families in Leaderboard Incentives, we fine-tune Qwen2.5 models from 0.5B to 14B parameters on GSM8k test items. We vary the dose (fraction of items, number of repetitions) and the form (verbatim or paraphrased). A matched control arm fine-tunes on an equal number of GSM8k training items, which is task-level benchmaxxing without item leakage.
The item-level prediction separates the two arms. Leakage should put the residual mostly on the seen items when the original test set is split into seen and unseen halves. Benchmaxxing should spread any residual evenly across items.
Map where the test is decidable. Finally, we chart the detectability frontier. This means the smallest contamination dose that yields $z_m > 2$ as a function of test-set size and of where on the accuracy scale the model sits. The result is guidance on when a replica benchmark can and cannot adjudicate contamination for a saturated benchmark.
Baselines
Raw drop (naive reference). The naive reference is the raw accuracy drop, original minus replica, as used by Zhang et al. It is run on the identical model set and items. Beating it means higher rank correlation with the injected dose, and fewer false positives on clean models when the replica is deliberately made harder or easier.
Log-likelihood detector (strongest existing competitor). The strongest existing competitor is the likelihood-based signal from the same paper. It measures a model's per-token log-likelihood of generating original test items, which is related to the near-zero perplexity signature reported in Studying and improving reasoning in humans and machines. The only reported number is its $r^2 = 0.36$ association with the GSM1k drop, from Zhang et al. No reported result was found on Lacuna for detection accuracy against known injected contamination, so we will measure that ourselves.
For fairness, both methods receive the same injected models and are scored against ground-truth dose by rank correlation and by area under the detection curve. The likelihood detector is allowed to use weights, which our method does not need. Beating it means matching it on verbatim leakage and exceeding it on paraphrased leakage, where likelihood of the exact text should be weak.
Ablations of the key ingredients. The key ingredient is the shared-shift correction plus its noise model. We isolate it with three ablations. The first uses raw-accuracy residuals without the logit transform, which tests the ceiling handling. The second uses residuals without the error bar, which tests the saturation guard. The third uses a line fitted on all models, including injected ones, instead of the clean reference set, which tests sensitivity to the reference.
Why not just…?
Why not just use the raw drop, as GSM1k did? The raw drop charges every model for any difficulty difference between the two sets. The ImageNet evidence shows such shared shifts are large and systematic. The decisive check is a synthetic difficulty shift: we build a slightly harder replica and apply it to clean models. The raw drop will flag all of them. Our residual should flag none. If the raw drop does just as well on this check, the line adds nothing and we will say so.
Why not just use likelihood-based contamination detection? This is a real competitor. On verbatim leakage with open weights, it may well win. It needs model weights, and it targets exact text, so it should weaken when leakage is paraphrased. It also has no natural way to account for replica difficulty. The paraphrase arm of the injection study settles this directly. If likelihood matches us there, our contribution narrows to closed-model auditing and to combining the two signals.
Why not just abandon static benchmarks for dynamic or live ones? DyVal and LiveBench prevent future contamination, but they say nothing about the static scores already published. DyVal also only covers formal domains. LiveBench explicitly cannot separate drift from difficulty, which is the decomposition this method provides. We test that claim by applying the fit to consecutive LiveBench task releases.
Proposed Experiments and Decisive Checks
The primary testbed is GSM8k paired with GSM1k, if GSM1k has been released under its stated June 2025 schedule. Otherwise we write a small difficulty-matched replica following the same protocol. Three secondary pairs broaden the test. DyVal difficulty tiers D1 to D4 serve as generated siblings. Consecutive LiveBench releases serve as temporal siblings. ImageNet and ImageNetV2 serve as a sanity check that our fitting code reproduces the reported $r^2 = 1.00$. For Winogrande, the benchmaxxing arm follows the Leaderboard Incentives protocol, including its Tune-before-Test variant, to see whether standardized pre-tuning removes task-level residuals.
Metrics are three:
- the Spearman correlation between $z_m$ and the injected dose;
- detection area under the curve on injected versus clean models;
- the false-positive rate under synthetic difficulty shifts.
Three results decide the proposal:
- Does the line exist? Clean models must fit the logit line with small scatter relative to sampling noise. If different model families form separate lines, the method needs one line per family and loses most of its power. That would be a negative result worth publishing.
- Does the residual separate leakage from benchmaxxing? Leaked items must carry the residual while benchmaxxed models spread it evenly. Otherwise the score measures benchmark-specific overfitting in general, and we will name it that.
- Does the saturation guard work? Near the ceiling, the error bar must prevent flags on clean saturated models while still catching injected ones above the detectability frontier.
References
- Measuring Robustness to Natural Distribution Shifts in Image Classification
- When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- Leaderboard Incentives: Model Rankings under Strategic Post-Training
- Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Studying and improving reasoning in humans and machines