When a model reports 85% on GSM8K, the number that matters is how much of that score comes from having seen these particular 1,319 problems during training. That is a property of a fixed, finite list of questions, not of some infinite pool of "math problems like these." We propose to estimate it directly, with an interval that stays valid however it is used. Define the inflation of a benchmark as the average, over every item, of the model's score on the published item minus its score on a verified clean counterpart: a rewritten version that has been checked to preserve the answer and to actually differ from the original. Clean counterparts are expensive to build. So we verify only a sample, drawn without replacement and steered by cheap contamination detectors. We then report a confidence sequence, meaning an interval that is valid at every stopping time, for the exact inflation on this benchmark. Its width shrinks to zero as the audit approaches the whole list.
The key property is that the cheap detectors are trusted only for efficiency, never for correctness. Memorization detectors are known to collapse on reasoning models, so this matters. If the detectors work, the audit stops after a small fraction of items. If they fail, the audit costs more but its error guarantee is unchanged. The mechanism comes from risk-limiting financial audits, which check a ledger against an AI risk score in the same way. We transplant it to contamination, where the "ledger" is a benchmark and the "misstatement" is score inflation.
Core Claim
Contamination detection has been framed as a yes/no question about a model: is the benchmark leaked? Leaderboards and model cards need something else, a corrected score with error bars for a named benchmark. Let the benchmark have $N$ items, let $y_i \in \{0,1\}$ be correctness on published item $i$, and let $\tilde y_i$ be correctness on its verified clean counterpart. The estimand is
$$\Delta = \frac{1}{N}\sum_{i=1}^{N} \left(y_i - \tilde y_i\right),$$
which is a fixed number, not a population parameter. We construct intervals $C_t$, built after $t$ verified items, that satisfy
$$\mathbb{P}\left(\forall t \le N:\ \Delta \in C_t\right) \ge 1-\alpha,$$
and that tighten using untrusted per-item side information $s_i$. This side information is cheap and available for every item: an automatic paraphrase confidence gap, a Min-k% style score, or output from a small proxy model. Item selection is weighted by $s_i$, and $s_i$ also enters as a control variate. The claim is that this auditor reaches a target interval width with substantially fewer verified items than uniform sampling when $s_i$ is informative. It should never undercover when $s_i$ is uninformative, including the post-reinforcement-learning regime where existing detectors are at chance.
Why This Intersection, and What Is Already Known
Three papers supply the statistical method. Confidence sequences for sampling without replacement builds Hoeffding- and empirical-Bernstein-type intervals for sampling without replacement. They remain valid under continuous monitoring and reach zero width when $n=N$; we use them as our unweighted baseline. Risk-limiting Financial Audits via Weighted Sampling without Replacement extends this to weighted sampling with betting-based martingales. Its sampling rule prop-MS draws items in proportion to a model's risk score. Its control variates let an accurate predictor shrink the audit while a useless one falls back safely. We reuse that machinery almost unchanged. Two things change: the audited quantity is a signed score difference rather than a monetary misstatement, and the risk score comes from contamination detectors. On Efficient and Statistical Quality Estimation for Data Annotation shows that finite-population acceptance sampling cuts manual inspection by up to 50% for label-error audits. It motivates treating datasets as finite lots. Its exchangeability assumption is also a caution: contamination clusters by subject and source, which is exactly why we sample with weights instead of uniformly.
Three papers motivate the problem. PaCoST compares a model's self-reported confidence on original and rephrased questions with a paired t-test. Its output is a significance verdict under a superpopulation model, with no correction for a finite benchmark, no valid early stopping, and no corrected score. ConStat estimates performance inflation, for example roughly 8.3% for Mistral-7b on GSM8K, by fitting a difficulty curve from about 20 reference models assumed clean. That assumption is the weak point the file itself flags: if the whole ecosystem has seen a benchmark, the reference curve moves. Our estimator uses no reference models. It compares the target model only to itself on verified counterparts.
One paper constrains the design. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that clean reinforcement learning after contaminated fine-tuning drops LiRA from 89.1% AUROC toward chance, and that contamination with chain-of-thought data leaves all ten detectors at 50–58% AUROC. Any design that trusts a detector's output is therefore ruled out. This is why detectors enter only as side information. It also sharpens what we measure. A chain-of-thought-contaminated model that generalizes to rephrasings shows a small $\Delta$ against paraphrases, so our counterparts must include structurally altered variants, such as changed numbers or isomorphic problems, and not only surface paraphrases.
Auditing the Audit finds that perturbation pipelines often produce "silent no-ops," where the "perturbed" input reaching the scorer is identical to the original. No-op rates can jump to 100% on some benchmarks. That is why each counterpart in our audit must be verified before it counts, and why verification, not model inference, is the expensive step being rationed.
Research Program
We will build an auditor with three parts. The first is a cheap screening pass over all $N$ items that computes side scores $s_i$. We will use the PaCoST paraphrase confidence gap when log-probabilities are available, an answer-only paraphrase gap for black-box APIs, and a proxy-model memorization score.
The second is a verification step for a sampled item. It generates a structurally altered counterpart and checks answer preservation, using a solver for math and human review for knowledge questions. It checks non-identity at the level of the text the scorer actually sees. It then scores the target model on the counterpart.
The third is the sequential estimator. It draws items without replacement with probability proportional to a mixture of $s_i$ and a uniform floor. It uses $s_i$ as a control variate and updates a betting confidence sequence for $\Delta$ after each verified item. Each difference $y_i - \tilde y_i$ lies in $\{-1,0,1\}$, so we shift it to $[0,1]$. That handles the two-sided errors which the RLFA paper lists as an open difficulty, because the estimand is a bounded mean rather than a one-sided total.
Two practical issues need engineering. The betting interval requires a grid search at each step, which the RLFA paper flags as a bottleneck; we will measure wall-clock cost and use a coarse grid with interval refinement. Clean counterparts also have their own noise: a verified rewrite can still be harder or easier than the original. We will therefore audit a second estimand, the same difference for a known-clean model released before the benchmark. This gives a difficulty offset that is estimated with the same sampled items, so no extra verification is needed.
Baselines
The naive reference point is exhaustive verification. Build and check a clean counterpart for all $N$ items and compute $\Delta$ exactly. This is the gold standard we use for ground truth, and its cost is $N$ verifications. Beating it means reaching a stated half-width, for example 2 points, with a small fraction of $N$ while holding coverage at $1-\alpha$.
The second baseline is uniform sampling without replacement with the empirical-Bernstein confidence sequence of Confidence sequences for sampling without replacement. It is the cleanest ablation of our key ingredient, because it has finite-population correction and anytime validity but no detector guidance. It will see the same items, verifier, and $\alpha$. If weighted sampling with control variates does not reduce the number of verified items needed to reach the target width when detectors are informative, the proposal's central contribution fails.
The third baseline is a fixed-size uniform subsample with a classical paired t-test interval, run with and without the finite-population correction factor $(N-n)/(N-1)$. It is what a practitioner would actually do. It is fair only when the sample size is fixed in advance, so we give it the oracle sample size, the one our method happened to stop at, and compare coverage and width.
The strongest existing method for magnitude is ConStat. It reports detecting 98.4% of sample-specific contamination in controlled trials and benchmark-specific inflation above 10% on the Open LLM Leaderboard. We will run it with its own reference-model protocol on the full benchmark. We will then rerun it after deliberately contaminating a share of its reference models, to measure how its estimate drifts. Beating ConStat means our interval covers the injected inflation in both conditions while ConStat's point estimate is biased in the second. On binary detection, PaCoST is the comparison, run on full benchmarks. The file reports only its significance threshold of $p<0.05$ and no AUROC, so no reported result was found on Lacuna for PaCoST detection accuracy, and we will measure it ourselves.
Why not just…?
Why not just take a random subsample of a few hundred items and run a t-test? For the binary question "is there any contamination," this probably works, and the finite-population correction is negligible when $n \ll N$. We concede that. The proposal does not compete there. It targets a corrected score with tight error bars on small benchmarks: HumanEval at 164 items, GSM8K's 1,319-item test set, and the competition sets used in the Fragility paper. On these, the sample fraction needed for a ±2-point interval is large, so the correction and valid early stopping matter. The fixed-subsample baseline above tests this directly. If it matches our method's width and coverage at equal verification cost, the finite-population contribution is empty and we will say so.
Why not just use ConStat? ConStat already estimates inflation and needs no verified counterparts beyond its rephrasings. Its validity, however, rests on clean reference models, and its rephrasings are not checked for silent no-ops of the kind Auditing the Audit documents. The contaminated-reference experiment above is designed to show where that dependence breaks. If ConStat stays unbiased even with half its references contaminated, the reference-free argument weakens to cost alone.
Why not just verify every item once and be done? If verification is cheap, as for math with an automatic solver, exhaustive verification is the right answer and our method is unnecessary. The proposal pays off when verification needs humans, which is the case for MMLU-style knowledge and professional questions, or when audits are repeated per model release. We will report the crossover point: the ratio of verification cost to screening cost above which the sampled audit is cheaper at equal error.
Why not just trust a better memorization detector? On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that every detector it tested is near chance after reinforcement learning or chain-of-thought contamination. A detector-only verdict is therefore unreliable on exactly the models people care about now. Our design treats a broken detector as a cost increase rather than an error.
Proposed Experiments
We create ground truth by contamination we control. We fine-tune open base models on known fractions (0%, 10%, 30%, 100%) of GSM8K, ARC-Challenge, and a set of MMLU subjects, with contaminated items clustered by subject to break exchangeability. Exhaustive verified counterparts give the exact $\Delta$. For reasoning models, we reproduce the two settings of the Fragility paper on AIME, AMC, GPQA Diamond, OlympiadBench, and Minerva Math: fine-tuning contamination followed by clean GRPO, and chain-of-thought contamination of DeepSeek-R1-Distill-Llama-8B. That paper reports pass@1 inflation of 8.8% and 11.7% in these settings, which our intervals should recover. WMDP serves as a post-cutoff benchmark that is clean by construction for older models, and is the null case.
Metrics are coverage of the confidence sequence across repeated audits, the number of verified items needed to reach a fixed half-width, the stopping time of the test "inflation at most τ," and the wall-clock cost of the betting search.
Three checks decide the question. First, on the chain-of-thought-contaminated reasoning model, where detectors sit at 50–58% AUROC, coverage must stay at $1-\alpha$, with verification cost no worse than uniform sampling plus a small margin. Second, on fine-tuned models where detectors work, weighted sampling must reach the target width with clearly fewer verifications than the uniform confidence sequence. Third, at sample fractions above roughly one half on small benchmarks, the finite-population intervals must be measurably narrower than the uncorrected t-interval at equal coverage. If the first check fails, the safety claim is false. If the second fails, side information adds nothing and the method reduces to existing finite-population confidence sequences.
References
- Confidence sequences for sampling without replacement
- Risk-limiting Financial Audits via Weighted Sampling without Replacement
- On Efficient and Statistical Quality Estimation for Data Annotation
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- ConStat: Performance-Based Contamination Detection in Large Language Models
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits