Contamination Hides in the Correlations: Copula Tests for Leaked Benchmarks After RL

Most tools that check whether a language model has seen a benchmark during training ask one question per test item: is the model unusually confident on it? That works for models trained only by next-word prediction. It stops working once a model goes through reinforcement learning (RL) post-training, the reward-driven fine-tuning that produces modern reasoning models. On The Fragility of Benchmark Contamination Detection in Reasoning Models contaminated models on competition-math test sets and found that a short phase of clean RL pushes reference-free detectors to about 50% AUROC (the area under the ROC curve, where 50% means coin-flip). Models contaminated with leaked chain-of-thought solutions fooled all ten detectors tested, which averaged 50–58% AUROC. The leak still improved accuracy by 11.7%, but the confidence gap the detectors rely on was gone.

This proposal tests a specific idea about where the evidence goes. RL may equalize the average level of each detection signal between seen and unseen items while leaving intact how those signals move together. The signals in question include token-level surprise, confidence in the answer, how similar repeated reasoning attempts are to one another, and how much confidence drops under paraphrase. A memorized item should be jointly extreme on several of these at once in a coordinated way, even when no single signal separates members from non-members on average. Copulas are the statistical tool built for this. A copula is the part of a joint distribution left over once each variable's own distribution has been standardized away, so it describes dependence and nothing else. We propose to fit a copula to detection signals on items known to be clean, and to flag items or whole benchmarks whose dependence pattern departs from it.

Core Claim

The claim is falsifiable. In the controlled contamination settings of the fragility paper, a test that uses only dependence between signals will detect contamination well above chance, specifically at item-level AUROC of 0.70 or higher in the chain-of-thought setting where existing detectors score 50–58%. It will do so even though each signal on its own has been made non-discriminative, because we deliberately rank-transform each signal against the clean reference set.

The motivating precedent comes from dependency modeling. Correlation Coefficients are Insufficient for Analyzing Spike Count Dependencies showed that variables can have zero linear correlation while remaining strongly dependent, and that assuming independence there distorts entropy by more than 25%. Copula variational inference showed that restoring dependence on top of independent per-variable factors changes predictive log-likelihood from −383.2 to −50.5 on a 100,000-node model. Our hypothesis is the detection analogue: per-signal gaps can vanish while the joint distribution still differs.

The fragility paper also constrains the design. It explains the cloaking as PPO-style clipping compressing the loss variance of unseen items. That mechanism acts on per-item loss levels, and it gives no reason to expect cross-signal dependence to be erased in the same way. It is still possible that dependence collapses too. If so, our experiments will show it, and that negative result would itself be informative.

Research Program

For each benchmark item we compute a vector of roughly twenty detection signals. These come from four sources.

First, token-level standardized log-probabilities in the style of Min-K%++. That paper normalizes each token's log-probability by the vocabulary mean and spread and averages the worst k%. Instead of collapsing the sequence to one number, we pool these scores into a small fixed number of position blocks. This borrows the block-grouping trick that BSTabDiff uses to tame thousands of genomic features, and it gives variable-length items a fixed dimensionality.

Second, the high-pass volatility statistic of AECA, which treats the token-loss sequence as a signal and measures its filtered volatility.

Third, answer confidence on original and paraphrased items, following PaCoST, which compares the model's probability of "Yes, this is correct" on original versus rephrased questions.

Fourth, the spread of sampled reasoning rollouts. Here we use both the minimum nearest-neighbor edit distance of Detecting RLVR Training Data via Structural Convergence of Reasoning and an embedding-space diversity measure.

The null model is fit on items known to be unseen. In controlled experiments these are held-out test items. In real audits they are post-cutoff or procedurally generated twins from LiveBench or A²RBench.

Fitting proceeds in two steps. Each signal is first mapped to its rank among the clean items, which gives it a uniform distribution on [0,1]. This step removes signal scale, intrinsic token rarity, and item difficulty insofar as they act on single signals. Sklar's theorem guarantees that what remains is the copula. Some signals, such as edit distances, are discrete, and discreteness makes the copula non-unique. We handle this with randomized tie-breaking and report how sensitive results are to it, since this is a known weak point of copula theory for discrete data. A truncated vine copula is then fit to the ranked signals of the clean items. A vine builds a many-variable dependence model out of a tree of two-variable copulas, as in Copula VI. We choose vines over a Gaussian copula because memorized items should be extreme on several signals simultaneously. That is tail dependence, and a Gaussian copula cannot represent it.

Write $u \in [0,1]^d$ for an item's rank-transformed signal vector and $c_0$ for the vine density fit on clean items. The item-level score is

$$s(u) = -\log c_0(u),$$

so items whose joint pattern is improbable under the clean dependence structure score high. For a whole benchmark, we run a two-sample test on the ranked signals of the suspect items versus held-out clean items, calibrated by permutation. This gives a p-value, as PaCoST does, but it is built from dependence rather than from mean shifts, and it decomposes into item-level evidence.

A diagnostic study runs alongside the detector. For each contamination stage, we measure separately how much of the member/non-member difference lies in individual signal distributions and how much lies in the copula. This directly tests whether RL moves the evidence from levels into dependence.

Baselines

The naive reference point is sequence loss or perplexity from a single forward pass. The fragility paper reports that loss-based AUROC drops by about 16% after clean RL and sits near 50% in the chain-of-thought setting. Every method, including ours, will be run on the same contaminated checkpoints, the same item splits, and the same held-out clean items.

The strongest likelihood-based competitor before RL is LiRA, a membership test that compares the model's loss to that of reference models trained with and without the item. The fragility paper reports 89.1% AUROC for LiRA on SFT-contaminated Qwen2.5-7B-Instruct, dropping by about 14% after GRPO. We will give LiRA the same reference-model budget it uses in that paper. On post-RL and chain-of-thought checkpoints, beating it means exceeding its post-RL AUROC on identical items.

The strongest reference-free likelihood detectors are Min-K%++ and AECA. On WikiMIA, Min-K%++ reports AUROC gains of 6.2–10.5 points over Min-K%, and AECA reports average AUC gains of 1.3–1.5 points over perplexity, Min-K%, and PAC. These two papers report relative gains only, so we will recompute absolute values ourselves.

The strongest method aimed specifically at RL-trained models is Min-kNN Distance. It reports an average AUC of 0.70 across 1.5B–32B models and 0.76 on OpenR1-Distill-7B, at about 6.65 seconds per item with 32 samples. We will give it exactly the same rollouts our method uses.

For benchmark-level audits, the baseline is PaCoST's paired t-test. It has no reported AUROC; on Lacuna, no reported result was found for PaCoST detection AUROC. We will therefore compare false-positive rates on clean benchmarks and power on contaminated ones at a matched significance level of 0.05.

Two ablations isolate the key ingredient. The first is an independence copula: the same ranked signals combined with Fisher's method, which assumes no dependence. It uses all the per-signal information we use but none of the dependence. The second is a Gaussian copula, which models dependence but has no tail dependence.

Beating the field requires two things. The vine score must exceed the best baseline's AUROC by at least 10 points in the chain-of-thought setting. It must also beat the independence-copula ablation by a clear margin, meaning non-overlapping bootstrap intervals over items.

Why not just…?

Why not just train a classifier on the concatenated signals? A gradient-boosted classifier trained on simulated contamination could also exploit dependence, and in-distribution it may match or beat the copula. This is a real threat, and we will run it. The copula's argument is that it needs only clean items, which auditors can obtain from post-cutoff data, whereas the classifier needs contaminated examples from a simulation that must match the unknown training recipe of the audited model. We will test this transfer directly: train the classifier on SFT-plus-GRPO contamination and evaluate on chain-of-thought contamination, and the reverse. If the classifier transfers as well as the copula, we will report the result as feature combination, not copula modeling.

Why not just make PaCoST's paraphrase comparison item-level? The difference between an original item and its paraphrase is one of our signals. The fragility paper's chain-of-thought result is the warning here. A model that has internalized the reasoning is equally confident on distributionally similar unseen questions, and close paraphrases are about as similar as unseen questions get. A single paraphrase gap should therefore collapse like the other signals. Its per-signal AUROC in our diagnostic study will show whether it does.

Why not just replace edit distance in Min-kNN with embedding distance? Better distances may lift rollout-diversity detection. However, diversity alone confounds memorization with easiness, because a clean, easy problem also yields near-identical solutions. The copula conditions diversity on the other signals rather than reading it in isolation. Embedding-based Min-kNN is included as a signal and run as a standalone baseline, so the comparison settles directly whether joint modeling adds anything beyond it.

Proposed Experiments

The central experiment reproduces both contamination stages from the fragility paper on AIME, AMC, GPQA Diamond, OlympiadBench, and Minerva Math. Stage one is SFT contamination of Qwen2.5-7B-Instruct followed by clean GRPO. Stage two is chain-of-thought contamination of DeepSeek-R1-Distill-Llama-8B. In both, half of each test set is leaked and half is held out, which gives exact item-level ground truth.

The first decisive check is the diagnostic decomposition. At every stage we report per-signal AUROC alongside a dependence-only test in which marginals are equalized by construction. The hypothesis fails if dependence-only detection is also near 50% after RL.

The second check is the ablation ladder: independence copula, then Gaussian copula, then vine. The vine must beat the independence copula clearly.

The third check is a no-clipping control. The fragility paper shows that removing clipping restores likelihood gaps. If our dependence signal survives exactly where clipping erases levels, that supports the mechanism.

Secondary experiments probe generality and cost. Short-item robustness is tested on 32-token WikiMIA splits, where AECA is known to fail. Cost is tested by repeating rollout-based signals with 8 instead of 32 samples. False-positive control is tested by running benchmark-level tests on clean models with WMDP, which The WMDP Benchmark released after their training. Finally, we audit open models on MMLU, HellaSwag, ARC, TruthfulQA, and WinoGrande and compare agreement with PaCoST's flags.

The result would matter either way. A positive result gives auditors a detector that survives RL, without needing labeled leaks. A negative result would show that RL erases contamination from per-item model behavior entirely, not just from its averages. That would strengthen the case for private or procedurally generated evaluation.