Set-Level Kernel Audits for Contamination That Survives Reinforcement Learning

When a language model has seen a benchmark during training, its high score may reflect recall rather than skill. The standard check scores each test question on its own, for example by how surprisingly likely the model finds its tokens. That per-question check has recently stopped working. Reinforcement-learning post-training of reasoning models erases the likelihood gap between seen and unseen questions. Ten standard detectors fall to 50–58% AUROC, close to chance, even though contamination still inflates scores by about 11.7% (On The Fragility of Benchmark Contamination Detection in Reasoning Models). Separately, contamination leaves a footprint inside the network: representations of leaked items collapse into a few shared directions (A Narrowing Geometry in Contaminated Reasoning). A shared direction is a property of a set of items. No single item shows it. This suggests asking about the whole benchmark at once: "do the model's internal representations of this benchmark's questions, taken together, look different from its representations of freshly generated twin questions of the same kind?"

The natural tool for that question is a kernel two-sample test. It compares two sets of points through all pairs of similarities between them. Comparing all pairs is the same all-to-all computation that limits interacting-particle simulations, and it costs time quadratic in the number of items. With large question banks, many layers, high-dimensional features and thousands of permutations needed for calibration, that cost is real. We propose to borrow the compression used to speed up those particle methods, kernel thinning, to shrink the audit. We also test a question the existing theory leaves open: does compressing the audit set throw away the statistical power needed to detect contamination?

Core Claim

The claim has two parts, and either can fail on its own terms.

The first part is about detection. After reinforcement learning, contamination remains detectable as a difference between two distributions of internal representations: those of benchmark items and those of matched clean twins. This holds even when every per-item likelihood statistic is at chance, because the contamination signal lives in a low-dimensional direction that is shared across items and not known in advance. A per-item score fixes in advance which direction it looks along. A test that compares distributions through all pairs of items does not have to.

The second part is about cost. Kernel thinning compresses each set to roughly $\sqrt{N}$ representative points, following Thinned Mean Field Langevin Dynamics. The audit then runs in near-linear time while keeping close to full-set power. Uniform subsampling to the same size loses power, as the information limit of When Is Benchmark Contamination Detectable? predicts.

Concretely, let $P$ be the distribution of a model's layer-$\ell$ hidden states (or low-rank gradient features) on suspect benchmark items, and $Q$ the same on twins. The audit statistic is the squared maximum mean discrepancy,

$$\mathrm{MMD}^2(P,Q) = \mathbb{E}[k(x,x')] + \mathbb{E}[k(y,y')] - 2\,\mathbb{E}[k(x,y)],$$

where $x,x'$ are drawn from $P$, $y,y'$ from $Q$, and $k$ is a kernel measuring similarity between two representation vectors. The cross terms are where the all-to-all cost sits. The function that achieves this discrepancy, called the witness function, also gives a per-item score. That score lets the audit point to which questions most likely leaked.

How This Relates to Prior Work

Three papers motivate the proposal by showing that the gap exists. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that GRPO clipping shrinks the variance of the loss on unseen items and collapses per-item detectors. It never tests statistics computed over sets. A Narrowing Geometry in Contaminated Reasoning finds the low-dimensional collapse through its Eigenspectrum Concentration Index (ECI) and Top-Subspace Suppression (TSS). However, it describes the geometry of a single model's leaked items and does not frame contamination as a calibrated hypothesis test against a clean null. It also does not report detection AUROC after reinforcement learning. Min-K%++ shows that training turns seen examples into local peaks of the model's likelihood. It scores each item in isolation, and that per-item signal is the one reinforcement learning erases.

One paper constrains the design. When Is Benchmark Contamination Detectable? models contamination as a sparse mixture and bounds detectability by $\alpha\rho\sqrt{m}$. Here $\alpha$ is the leaked fraction, $\rho$ is how separable seen and clean behavior are, and $m$ is the number of audited items. It also shows that small-sample audits with $m<50$ have near-zero power. The bound treats items as independent scalar observations. Our test still obeys a bound of the same form. The bet is that $\rho$ measured on full representations stays large after reinforcement learning, while $\rho$ measured along the likelihood direction does not.

Two papers supply the methods. Thinned Mean Field Langevin Dynamics compresses an interacting swarm to a kernel coreset of size about $\sqrt{N}$. This cuts cost from $O(N^2)$ to $O(N^{3/2})$ while preserving convergence, and it reports that uniform random subsampling adds variance that degrades accuracy. Particle-based Variational Inference with Preconditioned Functional Gradient Flow replaces pairwise kernel interactions with a learned vector field. We reuse that idea in a limited way: a small learned witness network serves as an $O(N)$ alternative to the kernel test.

For building clean twins we draw on two more sources. A²RBench generates fresh, verified items programmatically. ConStat uses rephrased reference variants. ConStat differs from our method in that it calibrates performance gaps across about 20 external reference models. We calibrate within one model against twins and so need no reference population.

Research Program

We first build controlled contamination where the ground truth is known. For the reasoning-model regime we reproduce the two-stage protocol of the Fragility paper on AIME, AMC, GPQA Diamond, OlympiadBench and Minerva Math. In Stage I a model is fine-tuned on half of each benchmark and then trained with GRPO for one or more epochs. In Stage II a DeepSeek-R1-Distill-Llama-8B model is contaminated with chain-of-thought solutions. For the mechanistic regime we reproduce the Narrowing Geometry setup on GSM8K and MSCalc with Qwen2.5 1.5B/7B and Llama-3.1 8B. In every setting the never-trained half of each benchmark is an exact clean null drawn from the same distribution. This lets us separate "the test detects leakage" from "the test detects that twins look different."

Next, we extract features and build twins. Features are mean-pooled hidden states at several layers plus random projections of per-item gradients. The gradient features are included because the Narrowing Geometry paper locates the signal in the coupling between activations and gradients. In realistic audits the untrained half is unavailable, so we compare three twin sources. The first is the held-out split, used as an oracle. The second is fresh programmatic regeneration: MSCalc instances from its generator, and A²RBench-style rule instances where applicable. The third is ConStat-style paraphrases. The main confound is that twins can differ from originals in surface style. We control for it by running the identical test on the model before contamination. A valid audit must accept the null there, and we report how far each twin source moves the statistic on the clean base model.

Then we scale the test. The full test computes the MMD with a Gaussian kernel whose bandwidth is selected on a held-out fold, and calibrates by permutation. The thinned test compresses each set with kernel thinning before testing. Its cost is near-linear in $N$ times the number of permutations, compared with quadratic for the full test. Finally, we measure power. We trace detection power as a function of coreset size $M$, leaked fraction $\alpha$ (set by varying how much of the benchmark is trained on), and number of reinforcement-learning epochs. We then check whether thinned audits track the full-set curve while uniform subsampling to the same $M$ falls off as $\sqrt{M}$, as the information-limit paper's bound implies.

Baselines

The naive reference point is per-item loss and Min-K%++ (Min-K%++). We aggregate these to the set level with the power-calibrated resampling audit of When Is Benchmark Contamination Detectable?, so that set-level comparisons are fair. The Fragility paper already reports these detectors near 50% AUROC after one GRPO epoch, and 50–58% averaged over 10 methods in Stage II.

The strongest per-item detector is LiRA, the likelihood-ratio membership attack. It reaches 89.1% AUROC before reinforcement learning and drops by about 14% after one GRPO epoch (On The Fragility of Benchmark Contamination Detection in Reasoning Models). Beating it means that our witness-function per-item AUROC exceeds post-GRPO LiRA on the same Stage I splits. At the set level it means that our test reaches higher power at a 5% false-positive rate, using the same models, the same items and no extra reference-model training.

The strongest geometric competitor is ECI from A Narrowing Geometry in Contaminated Reasoning. We give it the same permutation calibration against twins, which turns it into a covariance-spectrum two-sample test. No reported result was found on Lacuna for ECI as a post-reinforcement-learning detector with an AUROC.

ConStat (ConStat) serves as the black-box reference. It reports an estimated inflation of $\delta \approx 8.3\%$ on GSM8K for Mistral-7b. We run it with its own reference models and report its detections alongside our compute cost.

All methods share the same items, twin sets and permutation budget. Learned components, namely the kernel bandwidth and the witness network, get a tuning budget matched to the number of hyperparameter trials given to LiRA's shadow models.

Three ablations isolate the key ingredient. A linear kernel, which only compares means, tests whether anything beyond the mean matters. A covariance-only statistic tests whether anything beyond second moments matters. Rerunning the test after projecting out the top subspace that TSS identifies tests whether the signal really lives in the "memorization directions."

Why not just…?

Why not just use ECI with a threshold? This is the most serious threat. If contamination is only a variance collapse, a calibrated covariance test captures it at $O(Nd^2)$ cost, and the all-pairs kernel adds nothing. The covariance-only ablation settles this directly. If it matches the Gaussian-kernel MMD after reinforcement learning, we will report that the contribution reduces to a properly calibrated ECI audit with power analysis. That result is still new, since no file paper reports one, but it is not a relational signal. We expect the kernel test to pull ahead when reinforcement learning rotates or partially disperses the collapsed subspace, because a characteristic kernel is sensitive to all differences between distributions, not just their second moments.

Why not just use a linear-time MMD estimate or random features instead of kernel thinning? Both remove the quadratic cost more cheaply, and the file contains no evidence against them for audits. We run both at matched compute. The thinning paper's argument is that random compression adds variance that has to be paid for with more points. Our power-versus-$M$ curves test whether that variance penalty appears in the audit setting. If it does not, we recommend the simpler estimator and treat thinning only as the remedy for the expensive permutation-calibration loop.

Why not just aggregate per-item scores over the whole set? The power-calibrated audit already does this, and the $\alpha\rho\sqrt{m}$ bound shows why it fails after reinforcement learning. Averaging over many items cannot rescue a scalar whose separability $\rho$ is near zero. The baseline comparison measures exactly this.

Why not just use ConStat? It depends on clean reference models. Its own analysis concedes that this dependence breaks if the whole ecosystem trains on the leak. Our audit needs only the target model and twins, at the price of depending on twin quality. We quantify that cost with the clean-base-model null check.

Decisive Checks

The proposal succeeds if it passes three tests. First, on Stage I splits after GRPO, where per-item detectors sit near 50% AUROC, the kernel test rejects the null with high power at moderate leaked fractions. On the uncontaminated base model with the same twins, it holds its nominal 5% false-positive rate. Second, the test's statistic drops sharply once the TSS subspace is projected out, which ties the detected signal to the known mechanism and not to style differences between twins. Third, thinned audits at $M\approx\sqrt{N}$ stay within a small margin of full-set power while uniform subsampling to the same size does not. That would show compression preserves the effective sample size in the information-limit bound.

If the first check fails, contamination after reinforcement learning is not distributionally visible in these features, and that is a reportable negative result. If only the covariance ablation explains the gains, the claim narrows to calibrated ECI.