When a language model scores well on an old public benchmark, we cannot tell how much of that score it would have earned if it had never seen those questions, or close paraphrases of them, during training. This proposal estimates that missing "clean" score one item at a time. First, we learn how the model performs on questions it provably could not have seen. We use that to predict, for each old benchmark item, the probability that the model would answer it correctly without having seen it. That probability is a soft pseudo-label: a number between 0 and 1 that stands in for an unobservable outcome. We then compare the model's actual answers against these predictions. A statistical model sorts correct answers into two kinds: those that fit the model's general ability, and those that are "too good to be true." The result is a benchmark score with a confidence interval, an estimate of how much the score is inflated, and a per-item probability of contamination. The method needs only the model's sampled answers, not its internal token probabilities.
The opportunity comes from putting two literatures side by side. Contamination research has shown that leakage is often semantic rather than verbatim. Soft Contamination Means Benchmarks Test Shallow Generalization found semantic duplicates in the Olmo3 training data for 78% of CodeForces problems. It also showed that fine-tuning on such duplicates lifts MuSR accuracy by about 20%, even on unseen items. Separately, soft pseudo-labeling research has worked out when predicted probabilities can replace missing labels for evaluation. Pseudo-Label Validation for Unsupervised Domain Adaptation proves that when you score against soft pseudo-labels, the predictor's bias dominates the error while its variance averages out. It follows that the predictor should be deliberately undersmoothed, meaning regularized less than one tuned for predicting individual items. Our view is that contamination auditing is an instance of this same setting. The "source domain" is items that are certifiably fresh for the model. The "target domain" is old benchmark items, whose observed correctness has been corrupted by exposure. What we want to impute is the correctness the model would show without that exposure.
Core Claim
Our claim is that calibrated, undersmoothed soft pseudo-labels, fit on a model's own behavior on fresh items and combined with its observed answers through a two-component mixture, give an item-level estimate of clean accuracy. We expect this estimate to be more accurate than benchmark-level performance audits and than scoring on rephrasings.
The soft form matters for a concrete reason, and it is the key ingredient we isolate. The average of calibrated probabilities is an unbiased estimate of expected accuracy. The average of thresholded (argmax) pseudo-labels is not: an item the model would solve 60% of the time gets full credit. This bias does not average out across items, which is exactly the failure mode the oracle inequality in the pseudo-label validation work warns about.
The scoring rule is as follows. Let $y_i \in \{0,1\}$ be the model's observed correctness on benchmark item $i$. Let $p_i$ be the imputed probability of clean correctness. Let $q$ be the probability of a correct answer on a contaminated item, which is estimated. Let $\pi$ be the benchmark's contamination rate. Under the mixture model, the posterior probability that item $i$ is contaminated is
$$r_i = \frac{\pi\, q^{y_i}(1-q)^{1-y_i}}{\pi\, q^{y_i}(1-q)^{1-y_i} + (1-\pi)\, p_i^{y_i}(1-p_i)^{1-y_i}},$$
with $\pi$ and $q$ fit by expectation-maximization across items. The adjusted score uses the observation where it is trustworthy and the imputation where it is not:
$$\hat{A}_{\text{clean}} = \frac{1}{n}\sum_{i=1}^{n}\big[(1-r_i)\,y_i + r_i\,p_i\big].$$
The inflation estimate is the raw accuracy minus $\hat{A}_{\text{clean}}$. Averaging only the $p_i$ would also estimate clean accuracy, but it would throw away the real evidence carried by items judged clean. The mixture keeps that evidence and so reduces variance.
This is the same move STABLEVAL makes when it replaces majority vote with Posterior Expected Credit fit by expectation-maximization over annotators. STABLEVAL assumes the evaluation items are clean and models noise from human judges. We model a different corruption, the evaluated model's own exposure, and we need an imputed clean prior that STABLEVAL does not have.
Research Program
The build has three parts.
The first part is a fresh-item pool for each task family. The pool has the same answer format and a comparable difficulty range, and it must be released or generated after the audited model's training cutoff. We draw on LiveBench, which sources post-June-2024 tasks and scores them with deterministic scripts. We add procedurally generated items from controllable generators in the spirit of A²RBench and CrossWordBench. For GSM8K-style arithmetic, we also generate new multi-step problems with controlled step counts, following the MSCalc construction in A Narrowing Geometry in Contaminated Reasoning.
The second part is the imputer. It is a small, calibrated model that predicts the audited model's correctness from item-side descriptors that do not depend on whether the item was seen. These descriptors include reasoning-step count, constraint count, answer length, a frozen text embedding, and the solve rate of a panel of reference models. The imputer is trained on the model's sampled answers to fresh items, with several samples per item so that the training targets are themselves soft.
Because the fresh pool and the old benchmark differ in difficulty and topic, this is covariate shift, and the imputer is exposed to the bias problem the pseudo-label validation paper analyzes. Two design decisions follow. We sweep the imputer's regularization and test that paper's prediction: the regularization that minimizes error in the aggregate accuracy estimate should be weaker than the one that minimizes item-level log-loss. We also report how much the fresh pool and the benchmark overlap in descriptor space, since that paper's guarantees assume shared support.
The third part is the expectation-maximization scorer described above, with bootstrap intervals over items. An optional extension adds a per-item confidence gap between the original and rephrased question as a second observation channel, in the style of PaCoST. PaCoST uses that gap only in a dataset-level t-test. We use it only when log-probabilities are available, and the core method never requires them. This matters because closed APIs often withhold log-probabilities, which rules out PaCoST and the ordering test of Proving Test Set Contamination in Black-Box Language Models.
The central assumption is honest and testable: the fresh pool must really be clean for the audited model. If the pool is itself softly contaminated, the imputer inflates $p_i$ and the method underestimates contamination. Test of Time is a direct warning here. LLM-transformed questions erased the post-cutoff decay seen on cloze questions, where GPT-4o-mini dropped 3.83 points; on transformed physics questions, models scored 2.19 points higher after the cutoff. So "fresh" cannot mean "rephrased." We restrict the pool to items with new underlying structure, and we measure sensitivity by deliberately leaking a known fraction of the pool.
Baselines
The naive reference is raw benchmark accuracy, which amounts to assuming zero contamination. Any useful method must estimate the counterfactual clean accuracy more accurately than raw accuracy does. On contaminated models, raw accuracy's error is the true inflation.
The strongest existing competitor is ConStat. It fits a difficulty curve from reference models relating performance on the original benchmark to performance on a reference benchmark, then flags a model whose original score exceeds its predicted score. It reports a 98.4% detection rate on sample-specific contamination in controlled experiments, and inflation estimates such as δ ≈ 8.3% for Mistral-7b-v0.1 on GSM8K and δ ≈ 6.6% for Llama-3-70b on ARC. It is the strongest baseline because it answers the same question, "how much is this score inflated," without needing log-probabilities. We run it with the same audited models, the same reference-model panel our imputer uses for descriptors, and reference benchmarks built from the same fresh pool, so that both methods see identical clean-side information.
On detection, beating ConStat means matching its benchmark-level rate. On estimation, it means lower absolute error in the inflation estimate against known ground truth, especially in two regimes: low contamination rates, and cases where the reference panel is itself partly contaminated. ConStat is known to produce false negatives in the second regime. ConStat gives no item-level output, so for item-level comparisons we give it the strongest fair extension we can: per-item predicted correctness from its difficulty curve.
A second baseline is accuracy averaged over several semantic rephrasings of each item, the simplest "decontaminated" score. It gets the same number of generations per item as our method.
A third is a PaCoST-style per-item confidence gap, used only for models that expose log-probabilities and evaluated for item-level contamination detection.
Two ablations target the proposal's key ingredients. The hard-label ablation thresholds $p_i$ at 0.5 before scoring, which tests whether soft credit is what reduces estimation bias. The smoothing ablation uses an imputer tuned for item-level log-loss rather than undersmoothed, which tests whether the pseudo-label validation theory carries over. A third ablation drops the mixture model and averages $p_i$ directly, to measure the variance benefit of keeping the observations on items judged clean.
All methods get identical tuning budgets. Every hyperparameter is selected on a separate set of controlled-contamination runs that does not overlap the evaluation runs.
Why not just…?
Why not just report the model's score on a fresh benchmark like LiveBench? For overall ranking, that may well be enough. If the only question is "which model is better," our method adds little. The question here is different: how much of a specific old benchmark score is real, and on which items. A fresh leaderboard cannot answer that, and old benchmarks cover content and formats that fresh ones do not. We will test whether this distinction matters empirically. If our adjusted old-benchmark scores rank models almost exactly as LiveBench does and add no explained variance in held-out fresh performance beyond it, the aggregate score is redundant, and the contribution shrinks to item-level attribution.
Why not just score on rephrased versions of each item? Semantic contamination transfers to rephrasings. The Soft Contamination study found that fine-tuning on semantic duplicates raised MuSR accuracy by about 20% on seen and unseen items alike, and Test of Time showed that rephrasing launders temporal and influence-function signals. The attribution of a question to its source paper fell from 77.5% to 17.5% once the question was transformed. We therefore expect rephrase-averaged accuracy to remain inflated. Our controlled runs measure exactly how much, because ground-truth clean accuracy is known there.
Why not just use ConStat? ConStat is a real threat, since it already estimates inflation at the benchmark level. Our method differs in three ways. It learns from the audited model's own behavior on fresh items rather than depending mainly on a reference-model curve, which ConStat's authors note can shift if the reference panel is contaminated. It produces per-item posteriors. And it uses soft rather than thresholded correctness. If ConStat matches our aggregate inflation error in every regime, including a deliberately contaminated reference panel, then the honest conclusion is that our value is item-level attribution alone, and we will report it that way.
Why not just suppress memorization directions inside the network and re-evaluate? Top-Subspace Suppression, from A Narrowing Geometry in Contaminated Reasoning, recovers reasoning on roughly half of leaked samples and improves consistency with a clean base model by 11.03% over LNE-Blocking. It also needs the model's weights, and it changes the model being evaluated. We include it as a reference on the open models, where the question is whether its recovered accuracy agrees with our imputed clean accuracy. Agreement between two very different methods would be evidence that both track the same quantity.
Proposed Experiments
The decisive experiment uses controlled contamination with known counterfactuals. We fine-tune Olmo3-family models and Qwen2.5-1.5B/7B on mixtures that include semantic duplicates of chosen subsets of MuSR, GSM8K and ZebraLogic. Contamination rates range from the ecologically realistic 4 per 10,000 training points reported in the Soft Contamination study up to heavy leakage. For each run, we train a twin on identical data with the duplicates removed. The twin's accuracy on each item is the ground-truth clean counterfactual.
The primary metric is absolute error between $\hat{A}_{\text{clean}}$ and the twin's accuracy. Secondary metrics are the area under the ROC curve for item-level contamination (using $r_i$ against known leaked items), the calibration of $p_i$ on twin outcomes, and interval coverage.
The core claim is supported under two conditions. The soft mixture estimator must beat raw accuracy, rephrase-averaging and ConStat on inflation error at low contamination rates. The hard-label and item-tuned imputer ablations must each measurably increase that error. If soft and hard credit perform the same, the soft pseudo-labeling ingredient is unnecessary and the claim fails.
The second experiment stress-tests the central assumption. We leak 0%, 10% and 30% of the fresh pool into training and measure how estimation error degrades. We also deliberately contaminate the reference panel and ask whether our method degrades more gracefully than ConStat, which relies on that panel more heavily.
The third experiment applies the audit to public models on MMLU, ARC, HellaSwag and GSM8K. Here there is no ground truth, so we check consistency with published signals. These include ConStat's reported inflations for Mistral-7b-v0.1 on GSM8K and Llama-3-70b on ARC, and the p = 0.001 ARC signal for Mistral-7B from the ordering test. We then report per-item contamination maps that no existing method provides.
References
- Soft Contamination Means Benchmarks Test Shallow Generalization
- Pseudo-Label Validation for Unsupervised Domain Adaptation
- STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
- A Narrowing Geometry in Contaminated Reasoning
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- Proving Test Set Contamination in Black-Box Language Models
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- ConStat: Performance-Based Contamination Detection in Large Language Models