Detecting Leaked Benchmark Sources as Hidden Confounders

Benchmark contamination means a language model saw test questions, or their answers, during training. Current detectors try to decide this one question at a time, using how confident the model is or whether its accuracy drops on questions written after its training cutoff. Both signals are failing on modern reasoning models. A single round of reinforcement-learning post-training pushes likelihood-based detectors to near-chance accuracy. Simple rephrasing erases the drop in accuracy after the cutoff date. This note proposes a different unit of audit. Benchmark questions rarely leak one at a time. A whole contest, a whole arXiv paper, or a whole scraped benchmark file leaks together. If we treat each source as an "environment" in the sense of multi-environment causal inference, then leakage becomes a hidden variable shared by every item from that source. We can test for that hidden variable with statistics whose false-positive rate is controlled, even when no single item gives a usable signal.

The identification lever is a crossing that benchmarks already contain. Genuine skill is organized by topic, for example combinatorics, dynamic programming or electrostatics. Leakage is organized by source, for example one contest round or one paper. Most sources span several topics and most topics span many sources. So a model's extra success that follows source rather than topic, and that clean reference models do not share, is the signature of a leaked source. We would build and validate a source-level contamination test on this idea. We target the regime where per-item detectors have collapsed: models that were contaminated and then post-trained.

Core Claim

Contamination still raises accuracy after post-training has hidden it from likelihood detectors. On The Fragility of Benchmark Contamination Detection in Reasoning Models reports two facts about the same setting. Contaminating DeepSeek-R1-Distill-Llama-8B gives an 11.7% accuracy boost, and yet almost all ten detectors average 50–58% AUROC (area under the ROC curve, where 50% is chance). Our claim is that this leftover accuracy boost can be identified at the source level, with false positives calibrated at a chosen rate. We model correctness of model $m$ on item $i$ as

$$\text{logit}\,P(Y_{mi}=1) = \theta_m - b_i + \alpha_{m,t(i)} + \gamma_{m,s(i)},$$

where $\theta_m$ is the model's overall ability. $b_i$ is item difficulty, estimated from a panel of reference models whose training cutoffs predate the item. $\alpha_{m,t}$ is the model's extra skill on topic $t$. $\gamma_{m,s}$ is a model-specific effect of source $s$. A clean model should have $\gamma_{m,s}=0$ for every source once topic and difficulty are accounted for. A nonzero $\gamma_{m,s}$ that the reference panel does not share is evidence that source $s$ leaked to model $m$. The model is identifiable because topics and sources are crossed. If every source had a single topic, $\alpha$ and $\gamma$ would be confounded and no amount of data would separate leakage from specialization.

How the Causal Literature Shapes the Test

Three pieces of the heterogeneity literature play distinct roles here.

The first supplies the method. Detecting hidden confounding in observational data using multiple environments shows how to detect a hidden variable shared within an environment: pair two units from the same environment and use one unit's treatment as a stand-in for the hidden variable. Their test kept a 5% false-positive rate, and detection power rose sharply as the number of environments grew from 25 to 200. We adapt this pairing directly. For items $i$ and $j$ from the same source, $j$'s residual correctness serves as a stand-in for the source's exposure. We test whether it predicts $i$'s residual correctness after conditioning on both items' difficulty and topic. We test the parametric $\gamma$ model and this nonparametric pairwise test side by side. Their scaling result tells us to expect power to depend on the number of audited sources, not just the number of items.

The second supplies the framing. Under the independent-causal-mechanisms principle, each causal module is autonomous and shifts independently of the others (Kun Zhang et al.). Mechanism-shift detectors such as iSCAN and the confounding measures of Detecting and Measuring Confounding Using Causal Mechanism Shifts both ask which environments altered a mechanism. Here the mechanism is the map from difficulty to correctness, and contamination is a source-specific shift in it. The latter paper's confounding measures were near zero on unconfounded graphs. That result is the behavior we must reproduce on clean models.

The third constrains the design. Putting Causal Identification to the Test warns that a violated constraint may mean the environments themselves shifted together, not that the targeted assumption failed. The same paper notes that the independent-mechanisms assumption across environments cannot itself be tested. For us, the danger is that a source can carry a house style that some models handle well for legitimate reasons. AtCoder phrasing is one example. We handle this with the reference panel. A style effect that the panel shares is absorbed, and only a model-specific source effect counts. The paper's warning still means the test detects leakage relative to the panel. We state this limitation explicitly.

On the contamination side, the work closest to ours detects leaked items from inside or beside a single model, item by item. A Narrowing Geometry in Contaminated Reasoning finds a low-dimensional activation collapse on leaked GSM8K items. Detecting RLVR Training Data via Structural Convergence of Reasoning finds that prompts seen in reinforcement learning produce near-identical sampled solutions. Neither uses heterogeneity across sources as an identification instrument, and neither controls error rates at the source level. Can Large Language Models Learn Independent Causal Mechanisms? builds the independence principle into a model architecture to improve generalization, which is unrelated to auditing. Test of Time motivates the whole proposal. Chronological analysis is the special case where "source" means "month", and that case is confounded with drift in difficulty: models did 2.19 points better on post-cutoff generated physics questions. Our crossing replaces calendar time with source identity and controls for difficulty explicitly.

Research Program

We would build three things. The first is a source-annotated item pool. We tag LiveCodeBench problems (LiveCodeBench) by contest round and algorithmic topic. We tag RealMath/arXiv items by source paper and subject class. We tag the AIME, AMC and OlympiadBench items used in the fragility study by competition year and topic. For each pool we report how many sources span at least two topics, since this crossing density limits what is identifiable.

The second is the test itself. It has two parts. One is the parametric source-effect model above, fit with a likelihood-ratio test on $\gamma_{m,\cdot}$ and permutation calibration, where source labels are shuffled within topic. The other is the nonparametric within-source pairing test. Correctness can be replaced by any weak per-item signal, such as pass rate over samples, Min-K%++ score, Min-$k$NN distance, or the gap between original and rephrased versions. This lets us ask whether pooling across sources revives signals that fail item by item.

The third is a controlled contamination testbed. We follow the two-stage protocol of the fragility paper on open models: a Qwen2.5-7B base model at Stage I and DeepSeek-R1-Distill-Llama-8B at Stage II. We deliberately leak a random subset of sources in four forms: verbatim question–answer pairs, chain-of-thought (worked-solution) data, rephrased versions, and a leaked set followed by a clean GRPO (group relative policy optimization) phase, which the fragility paper calls a cloaking device. Ground truth is known at the source level, so we can measure true calibration and power.

Baselines

The naive reference point is a chronological or reference-gap check. For each source, it compares the audited model's accuracy with the panel's mean accuracy and flags sources where the gap is largest. It ignores topic, so it represents what auditors do today. It must be run on the same items and the same reference panel. We beat it if we match its power at a 5% false-positive rate while it shows inflated false positives on topic-specialized clean models.

The strongest likelihood competitor is LiRA, aggregated per source by averaging item scores. Min-K%++ (Min-K%++) is a reference-free variant aggregated the same way. The fragility paper reports LiRA at 89.1% AUROC at Stage I before GRPO. After one GRPO epoch it drops by about 14%, and post-reasoning-model detectors average 50–58% AUROC. Min-K%++ is reported to improve over Min-K% by 6.2–10.5% AUROC on WikiMIA. No reported result was found on Lacuna for any source-level aggregated version of these detectors, so we compute those ourselves under identical sampling budgets.

The strongest black-box competitor is Min-$k$NN distance, aggregated per source. It reaches an average item-level AUC of 0.70 in the structural-convergence paper. We give it the same 32 samples per item that our pass-rate signal uses, so compute is matched.

Two ablations isolate the key ingredient. Dropping the topic term $\alpha$ tests whether the topic–source crossing is what prevents false alarms. Replacing panel-estimated difficulty with the audited model's own difficulty estimates tests whether the reference panel is needed. Success means the full test keeps a nominal 5% source-level false-positive rate across all four contamination forms and both stages. It must also beat every aggregated baseline in source-level AUROC after the GRPO cloaking phase, which is where the fragility paper shows per-item detectors fail.

Why not just…?

Why not just average existing per-item detector scores over each source? This is the most serious threat, and we include it as a baseline rather than dismiss it. Averaging only helps when per-item signals carry information. After GRPO, likelihood signals sit near chance (fragility paper), so averaging recovers little. Averaged scores also move with a source's style and topic, so they need the same confounding control we propose. If averaged Min-$k$NN, with the crossing and panel controls added, matches our correctness-based test, we would report that the contribution is the identification design rather than the signal. The pluggable-signal experiment is built to show exactly that.

Why not just fit a standard mixed-effects model with a source random effect? This is close to our parametric arm, and we say so. What the causal framing adds is three things. It identifies the crossing condition under which source and topic effects can be separated. It requires conditioning on difficulty measured by an external panel. It supplies a nonparametric pairing test that needs no logistic link. The topic ablation shows how much a naive random-effects model without topic crossing over-flags.

Why not just move to live or procedurally generated benchmarks? These prevent future leaks, but they do not audit the static benchmark scores already reported. Their temporal signal also breaks under rephrasing: Test of Time erased LiveCodeBench's post-cutoff decay with variable renaming. DyVal (DyVal) also covers only formal graph-structured tasks. Our method applies to static benchmarks and treats live benchmarks as one more source of clean reference items.

Decisive Experiments

The first check is calibration on negative controls. We fine-tune clean models heavily on non-benchmark data from a single topic, such as combinatorics problems from outside the pools. The topic-aware test must stay at 5% false positives while the topic-ablated and naive gap tests exceed it. If the full test also over-flags, the crossing argument fails.

The second check is power under cloaking. At Stage II, after GRPO and with chain-of-thought contamination, we plot source-level AUROC against the number of audited sources, varied from 25 to 200 to mirror the hidden-confounding scaling result. We also plot it against items per source. Our test should improve with more sources while aggregated likelihood detectors stay flat near chance.

The third check is robustness to rephrased leaks. Leakage via rephrased questions defeats the temporal signals in Test of Time. It should still produce detectable source effects, because the accuracy boost survives rephrasing even when the surface form does not.

Finally, we apply the calibrated test to public open-weight models on LiveCodeBench and the competition math suite. We report which sources are flagged, with false-discovery-rate control. This is the only experiment whose ground truth is unknown, so we present it as an audit rather than as validation.