When a language model has seen a benchmark's test questions during training, its score on that benchmark mixes two things: what it can actually do, and what it remembers. Existing contamination tools mostly answer a yes/no question, "was this seen?", and the ones that output a corrected number do so with heuristics. We propose to estimate the number people actually want: the accuracy the model would have on this benchmark if it had never seen it. The method writes a fresh set of unseen items in the benchmark's style, lets the model answer them, and then reweights those answers so the fresh set statistically resembles the original benchmark. The reweighting uses classifier-based density ratio estimation, a standard trick in which a classifier trained to tell two datasets apart gives, through its odds, how much more likely each item is under one dataset than the other.
The key design choice is where the density ratio is computed. The naive combination of the two fields trains a classifier on the audited model's own signals and weights the contaminated benchmark. That breaks the assumption importance weighting needs: that the relationship between input and correctness stays the same across the two sets. Memorization changes exactly that relationship. We put the density ratio on the other side. It compares the content of benchmark items with the content of fresh items, using an encoder that is independent of the audited model. On the fresh, unseen items, the model's correctness really is its clean behavior, so the standard assumption holds. Because the method uses only the model's answers, it needs no log-probabilities or hidden states. It should therefore survive the reinforcement-learning post-training that erases the likelihood signals current detectors rely on.
Core Claim
Let $B$ be the benchmark and $F$ a fresh reference set of items known to be unseen, such as rephrasings or template variants. Let $z$ be an item's content representation from a frozen encoder that has nothing to do with the audited model, and let $c_j \in \{0,1\}$ be whether the audited model answers fresh item $j$ correctly. A calibrated classifier with output $s(z) = P(\text{item from } B \mid z)$, trained on balanced samples, gives the ratio $w(z) = p_B(z)/p_F(z) = s(z)/(1-s(z))$. The contamination-corrected accuracy is then the self-normalized weighted fresh accuracy
$$\hat A_{\text{clean}} = \frac{\sum_{j \in F} w(z_j)\, c_j}{\sum_{j \in F} w(z_j)},$$
and the estimated inflation is $\hat\delta = A_{\text{raw}}(B) - \hat A_{\text{clean}}$.
This estimate is consistent under one checkable assumption: for items the model has not seen, the probability of a correct answer depends only on the content $z$, and not on which set the item came from. Our claim is twofold. First, this estimator recovers clean benchmark accuracy with lower error than unweighted fresh-set accuracy, the heuristic DCR adjustment, and reference-model difficulty curves. Second, it keeps working after reinforcement-learning post-training, where token-likelihood detectors drop to chance.
Relation to Prior Work
Three papers supply the method.
- Bias Correction of Learned Generative Models via Likelihood-free Importance Weighting trains a real-versus-generated classifier and uses its self-normalized odds to debias Monte Carlo estimates. We reuse this estimator directly. Its full-support requirement becomes our main diagnostic: the fresh set must cover every region of content the benchmark occupies.
- Estimating Model Performance Under Covariate Shift Without Labels (PAPE) combines domain-classifier ratios with recalibration to estimate accuracy on shifted tabular data. We borrow the idea of estimating performance from ratios. We differ in having labels on the fresh side and in targeting text benchmarks, where the shift the paper assumes is broken on the benchmark side and intact only on the fresh side.
- Estimating the Density Ratio between Distributions with High Discrepancy using Multinomial Logistic Regression handles poorly overlapping distributions by inserting intermediate "bridge" distributions into one multi-class classifier. We use it when the fresh items are stylistically far from the originals, with graded-strength rephrasings serving as natural bridges.
Four papers motivate the problem.
- DCR: Quantifying Data Contamination in LLMs Evaluation is the only work that outputs an adjusted accuracy. It multiplies accuracy by one minus a fuzzy-logic contamination factor built from handcrafted probes. We replace that factor with an estimate that has a stated assumption and a consistency argument.
- ConStat: Performance-Based Contamination Detection in Large Language Models is our closest competitor. It corrects for the difference in difficulty between original and rephrased items by fitting a curve across about 20 reference models assumed clean. We correct the same difficulty difference item by item from content alone, removing the dependence on a clean reference ecosystem that ConStat itself flags as fragile.
- DICE: Detecting In-distribution Contamination in LLM's Fine-tuning Phase for Math Reasoning classifies contaminated items from the audited model's hidden states. It also shows that 2% in-distribution contamination produces a 10% absolute gain in benchmark score. DICE detects rather than corrects, and it needs the model's internal activations plus a clean twin model. Our method needs neither.
- On The Fragility of Benchmark Contamination Detection in Reasoning Models constrains the design. After reinforcement learning or chain-of-thought contamination, ten likelihood detectors fall to 50–58% AUROC while scores stay inflated. This rules out any design built on the audited model's probabilities, and it is why we use only correctness.
Research Program
We will build three components.
Fresh-set generators. For each benchmark we produce three kinds of fresh items:
- helper-model paraphrases, in the style of PaCoST;
- template variants with changed numbers or entities, following DICE's GSM-i-Syn and GSM-hard constructions;
- where possible, a small set of newly written items.
Content encoder and ratio classifier. The encoder is a frozen, independent sentence encoder augmented with cheap structural features such as length, number of reasoning steps, and number of answer options. The classifier is a logistic model or small MLP, calibrated on held-out data. It runs in binary mode by default and in multi-class bridge mode when the fresh set overlaps poorly with the benchmark.
Diagnostic layer. This layer reports:
- the effective sample size of the weights;
- the fraction of benchmark items with no well-matched fresh neighbor, which indicates a support failure;
- bootstrap confidence intervals on $\hat A_{\text{clean}}$.
We will also answer one analytical question: when does the "correctness depends only on content" assumption fail for unseen items? The likely failure is paraphrases that change the answer or the difficulty in ways the encoder cannot see. We will measure this directly using clean models, as described in the decisive checks.
Baselines
Raw benchmark accuracy. The do-nothing reference is the model's unadjusted accuracy on $B$. Any method must reduce the error to the true clean accuracy relative to this number. Under controlled contamination, that error equals the injected inflation. DICE reports about 10 points for 2% contamination, and the fragility paper reports an average pass@1 inflation of 8.8% in Stage I and 11.7% in Stage II.
Unweighted fresh-set accuracy. This is our estimator with all weights set to one. It is the ablation that isolates the density ratio, and it is run on exactly the same fresh items and model answers. Beating it means a lower absolute error to true clean accuracy, with the gain largest on template variants, whose difficulty is known to differ from the originals.
ConStat. This is the strongest existing competitor ConStat. It detected 98.4% of sample-specific contamination in controlled experiments, against 71% for the best baseline, and it estimates inflation directly, for example $\delta \approx 8.3\%$ for Mistral-7B on GSM8K. We run it with the same fresh items as its rephrased benchmark and a reference pool of openly available models of comparable size. We also report a reduced-pool setting, with a small number of reference models, and a partially contaminated pool, to test its dependence on clean references. On error in estimating clean accuracy, no reported result found on Lacuna for ConStat, so we measure it ourselves. Beating ConStat means matching its error with a full clean pool and clearly beating it when the pool is small or contaminated.
DCR. This is the only published adjusted-accuracy method DCR, which reports adjusted accuracies within 4% of uncontaminated baselines across 9 models. We reimplement its four probes and fuzzy combination on the same models. Beating it means mean absolute error below 4 points, and below DCR's own error on our controlled setups.
Audited-model features for the ratio. A second ablation computes the density ratio from the audited model's hidden states, in the manner of DICE's classifier, instead of from the independent encoder. We expect this variant to absorb contamination: the classifier learns "seen versus unseen" rather than content differences, which pushes the weighted estimate toward the inflated score. Confirming this supports the central design argument.
All methods get the same fresh items, the same answer budget from the audited model, and the same held-out tuning split.
Why not just…?
Why not just report accuracy on the rephrased set, without weights? This is the real threat. If paraphrases preserved difficulty, weighting would add variance and nothing else. The evidence says they often do not. ConStat exists precisely because rephrased and synthetic items differ in difficulty from the originals. DICE's GSM-hard numerical mutations change difficulty by construction. The unweighted ablation settles the question. If it matches our estimator on paraphrases, we will say so and restrict the claim to template variants and newly written items.
Why not just use ConStat? ConStat learns the difficulty difference from other models rather than from the items. That works only if a sizeable pool of clean reference models exists, a condition its authors call fragile once whole model ecosystems have been trained on the same web. Our estimator needs only the audited model's answers and an independent encoder. The reduced-pool and contaminated-pool experiments test whether this advantage is real or only theoretical.
Why not just move to procedural benchmarks? BeyondBench shows that models scoring over 90% on GSM8K can have accuracy cut in half or worse on freshly generated algorithmic tasks. Procedural generation is the right answer where it applies. It does not apply to knowledge benchmarks like MMLU, and it cannot audit scores that have already been reported on static sets. We use BeyondBench-style generated tasks as a sanity domain, where fresh items are exactly in-distribution and the weights should be near uniform.
Why not just use Min-K% or PaCoST? Both answer the detection question, not the correction question. PaCoST gives a dataset-level significance test. Min-K% and nine other likelihood detectors fall to 50–58% AUROC on reasoning models Fragility. We include them only to show the regime in which our correctness-based estimator still works and they do not.
Proposed Experiments
Controlled contamination with ground truth. We follow DICE and the fragility paper. We fine-tune open models, starting from Qwen2.5-7B-Instruct as in the fragility study, on a random half of the test items of GSM8K, ARC, MMLU, and HellaSwag, at duplication levels from one to ten. The untouched half gives the model's true clean accuracy on the benchmark distribution. The primary metric is absolute error between $\hat A_{\text{clean}}$ and that held-out accuracy, with bootstrap intervals and coverage of those intervals.
The reinforcement-learning regime. We repeat the above after one or more epochs of clean GRPO, and in the chain-of-thought-contamination setting on AIME, AMC, and GPQA Diamond. For reference, LiRA's AUROC falls from 89.1% there. Our estimator should keep its error roughly unchanged, because it never reads probabilities.
Real audits. We apply the method to Mistral-7B and LLaMA2-7B on MMLU and ARC. Proving Test Set Contamination found permutation-test evidence there at $p \approx 0.01$ and $p = 0.001$. We compare our $\hat\delta$ against ConStat's reported inflation figures where they exist.
Decisive Checks
The null calibration check comes first. On a model known never to have seen $B$, $\hat A_{\text{clean}}$ must match raw accuracy within its confidence interval. Failure here means the "correctness depends only on content" assumption breaks for that fresh-set type, and we drop that type.
Second, the weighted estimator must beat the unweighted one on at least the template-variant sets. Otherwise the density ratio earns nothing.
Third, the audited-feature ablation must drift toward the inflated score. That drift is what shows the design choice matters.
Fourth, when the effective sample size is low or the support diagnostic flags uncovered benchmark regions, bridge-mode ratios must reduce error relative to binary ratios. If they do not, the method should report "not estimable" rather than give a number.
References
- Bias Correction of Learned Generative Models via Likelihood-free Importance Weighting
- Estimating Model Performance Under Covariate Shift Without Labels
- Estimating the Density Ratio between Distributions with High Discrepancy using Multinomial Logistic Regression
- DCR: Quantifying Data Contamination in LLMs Evaluation
- ConStat: Performance-Based Contamination Detection in Large Language Models
- DICE: Detecting In-distribution Contamination in LLM's Fine-tuning Phase for Math Reasoning
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models
- Measuring Massive Multitask Language Understanding
- Proving Test Set Contamination in Black-Box Language Models