Auditing Benchmark Contamination as a Distribution-Free Two-Sample Test

Current tools for detecting whether a language model was trained on a benchmark ask one question per test item: does the model look unusually confident on this item? For reasoning models trained with reinforcement learning, that question has stopped working. After a model has seen chain-of-thought solutions to benchmark problems, it becomes about as confident on unseen problems of the same kind, and reinforcement learning compresses confidence scores further. Ten such detectors fall to roughly coin-flip accuracy in this setting. We propose asking a different question at the level of the whole benchmark: is the benchmark exchangeable with a freshly drawn control set of the same kind, as judged by the full distribution of cheap per-item behavioral signatures? "Exchangeable" means that no statistic could tell which items came from which set. A permutation test answers this question with exact false-alarm control. It needs no parametric assumption about score distributions and no fleet of reference models that may themselves be contaminated.

The link to work on Gaussian assumptions is specific. Existing detectors quietly assume that contamination shows up as a shift in the mean, or in a two-moment z-score, of a roughly bell-shaped score. Our hypothesis is that after reinforcement learning, what survives of contamination is a change in the shape of the benchmark's score distribution: a minority of memorized items forms a tail, and the bulk of the distribution is flattened. Tests that look at the whole distribution should keep power where mean-gap tests lose it. The note proposes to test this hypothesis, and it includes an ablation that could refute it.

Core Claim

In reasoning models that have gone through reinforcement learning, per-item membership detection is close to ill-posed, because contamination generalizes. Benchmark-level contamination, defined as behavior on the benchmark that does not transfer to matched fresh items, remains detectable. The detector is a permutation two-sample test over multivariate per-item signatures computed from one forward pass plus a handful of samples. It calibrates its own null through the permutations, and it gains power by being sensitive to tails and shape rather than only to means. The claim fails if a plain difference-in-mean-accuracy test between benchmark and control has the same power.

Motivation and Prior Work

Three papers motivate the proposal. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that PPO-style clipping in GRPO and RAFT++ shrinks the variance and raises the confidence of non-member samples. As a result, LiRA falls from 89.1% AUROC by about 14 points after one epoch of GRPO, and all ten detectors average 50–58% AUROC once distilled reasoning models are contaminated on chain-of-thought data. That paper diagnoses the failure but proposes no test that survives it. Test of Time shows that temporal-cutoff signals are artifacts of verbatim phrasing: rephrased arXiv questions show no post-cutoff decay (+2.19 pp on physics), and influence-function hit rates fall from 77.5% to 17.5%. This tells us which controls not to use: a paraphrase of a leaked item is not a clean control. Detecting RLVR Training Data via Structural Convergence of Reasoning finds that trained prompts produce low-diversity reasoning trajectories (Min-kNN Distance, average AUC 0.70). Its cost is about 6.65 seconds per item, and it gives no error-rate guarantees. We reuse its signal in a much cheaper few-sample form as one coordinate of the signature.

Two lines of work supply method. ConStat already defines contamination as non-generalizing performance and tests it against a rephrased or synthetic reference benchmark. Its null, however, comes from a hardness-correction curve fit across 20+ reference models, and ConStat itself found several of those models contaminated (for example, Mistral-7b-v0.1 on GSM8k with δ≈8.2%). We keep ConStat's definition but replace the cross-model curve with a within-model permutation null. Testing For Distribution Shifts with Conditional Conformal Test Martingales gives distribution-free, finite-sample false-alarm control against a fixed reference set. Its authors note that their test only bets on shifts in the mean of conformal p-values, and that is exactly the blind spot that clipping exploits. Our setting is batch rather than sequential, so a permutation test suffices; the conformal-martingale version becomes a later extension for auditing streaming leaderboards.

The distributional-assumptions literature constrains the design and explains it; it does not give us a theorem to import. Min-K%++ standardizes each token's log-probability by the vocabulary mean and standard deviation, which is a two-moment normalization. Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimization shows, for convex models, that a score can match Gaussian predictions in average error while its distribution departs from Gaussian in shape. We borrow the lesson that "matched first two moments" does not mean "same distribution". We do not borrow the theorem, which covers convex ERM rather than transformers, and we say so.

Research Program

The central object is a per-item signature vector. For each benchmark item we compute, from a single forward pass over the prompt and reference solution, the sequence loss, the Min-K%++ score, and the full quantile profile of per-token z-scores rather than only the mean of the lowest k%. From four sampled completions we add correctness and a cheap trajectory-diversity measure. The four-sample count is set so that the cost stays below one second per item. For a benchmark $B$ and a control set $C$ drawn from the same generating process, the null hypothesis is that signatures from $B$ and $C$ are exchangeable. The test statistic is the energy distance between the two signature samples, and we also run a tail-weighted rank statistic that compares upper quantiles. Recomputing either statistic under random relabelings of $B\cup C$ gives an exact p-value.

The reason to expect shape sensitivity to matter can be stated as a mixture. If a fraction $\varepsilon$ of benchmark items has been memorized, the benchmark's score distribution is

$$F_B = (1-\varepsilon)F_0 + \varepsilon F_1,$$

where $F_0$ is the control's score distribution and $F_1$ is the distribution for memorized items. A mean-difference test sees only a shift of size $\varepsilon\Delta$, where $\Delta$ is the gap between the means of $F_1$ and $F_0$. The number of items it needs therefore grows like $1/(\varepsilon\Delta)^2$; this follows from the standard error of a sample mean. Clipping shrinks $\Delta$ by pulling $F_0$ upward. Clipping does not necessarily remove the tail mass that $F_1$ places beyond the support of $F_0$, and a tail statistic can detect that mass with far fewer items when $\varepsilon$ is small. We will first check empirically whether post-RL member and non-member distributions actually differ in shape. The first experiment below does this, and if the answer is no, the project reduces to a calibrated mean test.

Control sets come from three sources, in decreasing order of cleanliness. The first is held-out splits in controlled injection experiments, where exchangeability holds by construction. The second is items published after the training cutoff from LiveCodeBench and RealMath/arXiv, used in their original form and never as paraphrases of leaked items. The third is number- and entity-perturbed variants of GSM8k-style items whose answers change. For the second and third sources, exchangeability is only approximate. We therefore measure the false-alarm rate empirically on models that were released before the benchmark or control existed, and we report this rate rather than assume it.

Baselines

The naive reference point is a two-sample test on mean accuracy between benchmark and control, using the same control sets and the same permutation calibration. It represents what an auditor would do with no distributional thinking. It is also the most serious threat, because the 8.8% and 11.7% accuracy inflations reported by the Fragility paper are large. Beating it means higher detection power at a false-alarm rate of 5% when the contaminated fraction or the benchmark size is small.

The strongest likelihood-based competitors are LiRA and Min-K%++, as implemented in the Fragility paper's protocol. The Fragility paper reports 89.1% AUROC for LiRA before GRPO, a drop of about 14 points after one GRPO epoch, and 50–58% average AUROC across detectors in Stage II. To compare on equal terms, we turn each of these per-item scores into a benchmark-level test by applying the same permutation machinery to that single score. Any gain can then be attributed to the richer signature and the shape-sensitive statistic, not to the aggregation step.

The strongest sampling-based competitor is Min-kNN Distance, which reports an average AUC of 0.70 and 0.76 on OpenR1-Distill-7B. We run it at its published 32 samples and also at our 4-sample budget, and we report wall-clock time next to power. The strongest set-level competitor is ConStat, which reports detecting 89% of syntax-contaminated and 98% of sample-contaminated models. We run it with its reference fleet on the same injected models, so both methods are scored on the same contaminated and clean checkpoints.

Two ablations isolate the key ingredient. The first applies a mean-only statistic to our full signature. The second applies our distribution-level statistic to the loss alone. If the mean-only statistic on the full signature matches the full method, the shape claim is wrong and the contribution reduces to a better choice of features.

Why not just…?

Why not just compare mean accuracy on the benchmark against fresh items? This alternative might work as well as our method, and our design treats it as the main baseline. If contamination always inflates accuracy by several points on large benchmarks, a mean test will be enough. Our claim is narrower: in the regimes that matter in practice, namely partial leakage, small benchmarks such as AIME or GPQA Diamond, and clipped RL models where accuracy gains partly transfer to controls, the shape of the distribution carries information that the mean does not. The sweep over contaminated fraction and benchmark size described below settles this question. If there is no region where our method wins, we will report that result.

Why not just train a classifier on token-level features from contaminated models? A supervised detector needs labeled contaminated models from the same training pipeline as the audited model. The Fragility paper shows that detector behavior depends heavily on the post-training algorithm (clipping in GRPO versus RAFT++). A classifier fit to one pipeline therefore has no guaranteed false-alarm rate on another, while a permutation test is calibrated on each audited model separately. We include a supervised logistic detector trained on the GRPO pipeline and tested on RAFT++ to measure how well it transfers.

Why not just run Min-kNN Distance, which already works on RLVR models? It is a real signal, and we incorporate it. Its limitations are its cost of about 6.65 seconds per item and its lack of error guarantees. If our cheap 4-sample diversity coordinate turns out to carry nearly all of the power, the honest conclusion is that a calibrated, cheap Min-kNN is enough. The per-coordinate ablation reports this directly.

Why not just use ConStat? ConStat shares our definition of contamination. Its null, however, rests on a reference fleet that it itself found partly contaminated. We will test whether ConStat's decisions change when the known-contaminated members of the fleet are removed, and whether our within-model test agrees with ConStat on the Open LLM Leaderboard models it flagged.

Proposed Experiments

The first experiment checks the premise. Following the Fragility Stage I protocol, we inject a random subset of AIME, AMC, OlympiadBench, Minerva Math, and GPQA Diamond items into SFT and then run clean GRPO and RAFT++. We compare the full distributions of member and held-out scores before and after RL, using quantile–quantile plots and energy distance. If clipping matches the distributions in shape as well as in mean, we stop and report the negative result.

The second experiment measures power and calibration. Using Stage I and Stage II (chain-of-thought contamination of distilled models such as DeepSeek-R1-Distill-Llama-8B), we vary the contaminated fraction from 5% to 100% and the benchmark size from 30 items (AIME scale) to several hundred. We report set-level power at a 5% false-alarm rate and the empirical false-alarm rate on uncontaminated twin checkpoints, for all baselines and ablations under matched compute.

The third experiment tests transfer to real audits. We apply the method with post-cutoff LiveCodeBench and RealMath controls and perturbed GSM8k and ARC controls to public models, measure false alarms on models released before each benchmark, and compare the resulting verdicts with ConStat's on the models it flagged. The decisive outcome is whether some region of contaminated fraction and benchmark size exists where the distribution-level test detects contamination that both the mean-accuracy test and the per-item detectors miss, with false-alarm rates that are measured rather than assumed.