When a language model has seen a benchmark's test questions during training, its score on that benchmark measures recall rather than skill. Current detectors mostly compare the model on the original question and on a rephrased copy, and flag contamination when the rephrased score drops. That comparison has a built-in confound. Rephrasing also changes difficulty, so an honest model drops too. The best existing fix calibrates the expected drop against "clean" reference models, and it breaks when every available model has seen the benchmark. We propose a different way to separate difficulty from memorization. Each benchmark item gets a graded ladder of meaning-preserving perturbations, from cosmetic reformatting up to structural rewrites. We then measure how the model's error rises as items move up the ladder, and extrapolate that curve back to the unperturbed original. An honest model's original-item error should sit on its own curve. A contaminated model's original-item error falls well below the extrapolation, and the size of that gap estimates how much the score is inflated.
The idea borrows its core move from risk extrapolation. That line of out-of-distribution work treats the per-environment error ("risk") as a function over training environments and asks what happens along affine directions outside the observed ones (Out-of-Distribution Generalization via Risk Extrapolation (REx)). Here the environments are perturbation levels. The original benchmark is the one environment we distrust, and we predict its risk from the others. Put in two sentences: memorization shows up as a kink in a model's error-versus-perturbation curve exactly at the canonical wording. Estimating the curve's shape on fresh items that the model cannot have seen replaces the need for clean reference models with a need for clean reference items, which are cheap to generate.
Core Claim
Fix a model $m$ and a benchmark $B$. Let $R_m(s)$ be the model's error on $B$'s items after perturbation at strength $s \in \{0, 1, \dots, K\}$, where $s=0$ is the canonical item. Let $\tilde R_m(s)$ be the same quantity on a matched set of fresh items that are procedurally generated or postdate the training cutoff, so they are guaranteed unseen. We assume that, absent contamination, benchmark items differ from fresh items by a difficulty offset rather than a different response to perturbation. The fresh items then give the model's own clean dose-response shape. We fit $R_m(s) \approx a + \tilde R_m(s)$ on levels $s \ge s_{\min}$ only, extrapolate to $s=0$, and define the inflation estimate
$$\Delta_m(B) = \big(\hat a + \tilde R_m(0)\big) - R_m(0),$$
where $\hat a$ is the fitted offset. A positive $\Delta_m(B)$ means the model does better on the canonical wording than its own perturbation behavior predicts. Significance comes from a bootstrap over items. Because the score uses only final-answer correctness, it works for black-box API models with no log-probabilities. It also works for short multiple-choice answers, a case where probability-based detectors are known to struggle (PaCoST).
A second claim concerns evasion. A developer who trained on paraphrased benchmark copies will show a flat curve through the paraphrase levels, followed by a drop at the structural levels. Fitting the offset on high levels and checking residuals at every lower level turns this into a detectable change-point rather than a blind spot. That blind spot is the specific weakness of plain rephrasing tests noted by ConStat.
Relation to Prior Work
Two detectors motivate the proposal and are its main competitors. PaCoST runs a paired t-test on the model's elicited confidence for original versus rephrased items. That makes it a one-rung version of our ladder with no difficulty correction, and it needs confidence elicitation. ConStat corrects for difficulty by fitting a translation curve from original to perturbed benchmark scores across reference models. It flags models whose drop exceeds that curve, and its authors acknowledge the method depends on those reference models being clean. We keep ConStat's insight that the rephrased score must be corrected for difficulty, but we take the correction from the same model's behavior on fresh items. This matters exactly when a whole generation of models is exposed. The survey Benchmark Data Contamination of Large Language Models reports that EvoEval rewrites of HumanEval cut scores of 51 models by 39.4% on average. We read that as evidence of the confound we target: a drop that large must mix memorization with difficulty change, and a single-rung comparison cannot apportion it.
Risk extrapolation supplies the method and also a warning. REx extrapolates risk beyond the training environments, but it notes that environments with unequal irreducible noise corrupt the variance signal. In our setting, the corresponding hazard is a perturbation that silently changes an item's correct answer. Such a perturbation raises error for reasons unrelated to memorization. We therefore require label preservation to be checked at every rung, as described below. An Invariant Learning Characterization of Controlled Text Generation builds synthetic environments by deliberately corrupting inputs (EVIAN). We reuse the idea of constructed environments, but we apply it to evaluation rather than to training a toxicity classifier.
The fresh-item side draws on existing contamination-limited sources. DyVal generates graph-structured reasoning items with controllable complexity and description order, and shows that order flips alone cost GPT-4 13.6% on reachability. That both supplies structural perturbation operators and shows they carry real difficulty that must be modeled. LiveBench and CausalProbe-2024 provide post-cutoff items. ABCD: All Biases Come Disguised documents positional biases in multiple-choice formats. This constrains us: option shuffling is not a free "zero-difficulty" rung, so its effect is measured on fresh items like every other rung.
Research Program
The first component is the perturbation ladder. For each benchmark we define five rungs. Rung 0 is canonical. Rung 1 is surface reformatting: answer-option reordering, template and whitespace changes. Rung 2 is lexical paraphrase. Rung 3 renames entities and swaps numbers, with answers recomputed. Rung 4 is a structural rewrite, for example reordering the premises of a GSM8k problem in the manner of DyVal's description orders, or recasting an ARC question into a different question form. Where answers can be recomputed programmatically (GSM8k, DyVal-style tasks) we recompute them. Elsewhere a two-model agreement filter is applied, followed by human spot-checks of 200 items per rung, and we report the label-flip rate per rung.
The second component is matched fresh items. The same ladder is applied to them: DyVal-generated items for arithmetic and logic, LiveBench math and reasoning items for competition-style math, CausalProbe-2024 for causal QA, and WMDP for knowledge QA in models trained before its May 2024 release, following PaCoST's use of it as a guaranteed-clean set. We study how well the "offset only" assumption holds and when it needs a slope term, by testing it on pairs of fresh sets that are both clean.
The third component is the estimator and test. It covers the offset fit, the extrapolation to rung 0, residual change-point checks at rungs 1 to 3, and bootstrap confidence intervals. It also includes a per-item variant, which asks whether a given item's canonical correctness is anomalous given its perturbed correctness, so that leaked subsets can be localized.
Baselines
The naive reference point is the raw rung-0-minus-rung-2 accuracy drop with a bootstrap threshold. This is what most practitioners compute informally, and our method must beat it by a wide margin or the ladder is not earning its cost.
The strongest existing method is ConStat. In its controlled synthetic study it reports detecting 98.4% of sample-specific contamination, against 71% for the best prior baseline, and it estimates real inflations such as about 6.6% for Llama-3-70b on ARC and about 8.3% for Mistral-7b-v0.1 on GSM8k. It is the strongest competitor because it is also black-box and performance-based, and it already corrects for difficulty. For fairness it receives the same perturbed benchmark (our rung 2 and rung 4 versions as its reference benchmarks) and the same number of model queries. It is run in two conditions: with genuinely clean reference models, and with reference models that we have deliberately contaminated. Beating it means matching its detection rate within noise in the first condition. In the second condition, it means exceeding it by at least 15 points of detection rate at a 5% false-positive rate, and producing an inflation estimate whose error is under half of ConStat's.
PaCoST is run on open models with its P(True) prompt and the same rung-2 rephrasings. No reported result was found on Lacuna for PaCoST's numeric detection accuracy; the file gives only its $p<0.05$ flags. Min-K%++ reports a 6.2 to 10.5 point AUROC gain over Min-K% on WikiMIA. It is included as the white-box membership-inference reference on open models, using identical items.
Two ablations isolate the key ingredient. The first uses a single rung with fresh-item correction. The second uses the full ladder without fresh-item correction, fitting the extrapolation shape directly on the benchmark's own rungs. If the single-rung variant matches the full method, the ladder is unnecessary. If the uncorrected ladder matches it, the fresh reference items are unnecessary.
Why not just…?
Why not just use PaCoST-style paired rephrasing tests? They answer "did the score drop?" but not "did it drop more than difficulty explains?" The 39.4% average EvoEval drop across 51 models (survey) shows that rewrites move scores for many models at once, which predicts high false-positive rates on clean models. Our planted-contamination experiments measure this directly, by reporting PaCoST's and the raw drop's false-positive rate on models fine-tuned only on unrelated data.
Why not just use ConStat? When clean reference models exist, it may well match us, and we say so. That condition is a parity check, not a claimed win. The case for our method rests on the all-references-contaminated condition, which ConStat's own authors flag as its failure mode. If ConStat survives contaminated references, for example because the translation curve is robust to uniform inflation, our contribution shrinks to black-box change-point detection of paraphrase-trained evasion. We would report that outcome.
Why not just evaluate on post-cutoff or live benchmarks like LiveBench? That tells you how good a model is now. It does not tell you whether a specific published score on GSM8k or MMLU was inflated, and it covers only programmatically gradable tasks. It also expires as new web dumps absorb the items (LiveBench). We use such items as the clean reference that makes auditing old benchmarks possible, so the two are complementary. The relevant test is whether our inflation estimate predicts a model's score gap between a static benchmark and its fresh counterpart, as in the COPA-versus-CausalProbe collapse reported by Chi et al..
Why not just report the rung-4 accuracy as the "honest" score? Rung-4 accuracy measures ability on harder, rewritten items, not what the canonical score would have been. It therefore understates clean models and cannot compare benchmarks. The uncorrected-ladder ablation and the clean-model calibration check show how much this shortcut misestimates known inflation.
Proposed Experiments
The decisive experiment uses planted contamination with known ground truth. Open models from the Llama, Mistral, and Qwen families are fine-tuned on random halves of GSM8k, ARC-Challenge, MMLU (Hendrycks et al.) and HellaSwag. Contamination is planted at three doses (1, 3 and 10 epochs) and in three forms (verbatim question and answer, question only, and paraphrased copies). Clean controls are fine-tuned on matched-size unrelated data. The held-out half gives the true inflation, measured as the accuracy difference between seen and unseen halves. We report detection rate at 5% false positives, AUROC, and absolute error of the inflation estimate. The reference-model condition for ConStat is varied as described above.
The first natural check applies the method to public checkpoints on the same benchmarks, together with WMDP for pre-May-2024 models as a guaranteed-negative control. A flag on WMDP counts as a false positive. The estimates are compared with ConStat's published inflations for Llama-3-70b on ARC and Mistral-7b-v0.1 on GSM8k. The second natural check correlates our estimated inflation on COPA and e-CARE with each model's drop on CausalProbe-2024.
Three results would falsify the approach. The first is if fresh-item dose-response shapes fail to transfer to benchmark items in clean models, meaning a large nonzero $\Delta$ with no contamination. The second is if label-flip rates at high rungs are large enough that fitting on those rungs dominates the estimate. The third is if the single-rung ablation performs as well as the full ladder. Any one of these would tell us that the curve-extrapolation idea, rather than simply more perturbation data, is not what drives detection.
References
- Out-of-Distribution Generalization via Risk Extrapolation (REx)
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- ConStat: Performance-Based Contamination Detection in Large Language Models
- Benchmark Data Contamination of Large Language Models: A Survey
- An Invariant Learning Characterization of Controlled Text Generation
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- ABCD: All Biases Come Disguised
- Min-K%++: Improved Baseline for Pre-Training Data Detection from Large Language Models
- Measuring Massive Multitask Language Understanding