Auditing Partial Benchmark Contamination with Tail-Risk Statistics

A reported benchmark score is only meaningful if the model has not effectively seen the answers. In practice, leaks are rarely all-or-nothing. A few hundred items get copied into a fine-tuning mix, or a search agent finds a dataset mirror for about 3% of questions, as Search-Time Data Contamination observed on HuggingFace. On those items accuracy jumps: that paper reports Sonar Deep Research at 37.5% on search-contaminated Humanity's Last Exam items versus 17.4% on clean ones. Two families of detectors are available, and both handle this partial regime badly. Detectors that check whether the model assigns unusually low loss to an item stop working after reinforcement-learning post-training. Detectors that compare a model's score on the original benchmark with its score on an equivalent variant look at the average gap, and an average spread over thousands of items dilutes a large gain concentrated on a few percent of them.

We propose to borrow a lesson from risk-sensitive retrieval. In that field, search systems are judged by their worst per-query regressions against a baseline, not only by their mean. We apply the same idea to contamination auditing. For every item we compare the model's result on the original question with its result on a meaning-preserving variant, correct that difference for how much harder the variant is, and then examine the upper tail of the corrected differences instead of their mean. The claim is that a tail statistic detects contamination affecting a small fraction of items with much more power than mean-based detection. It also points to which items are leaked. Because it uses only right/wrong outcomes, it should survive the post-training that hides loss-based signals.

Core Claim

Contamination produces a characteristic per-item signature: the model is right on the original and wrong on a variant more often than its ability predicts. For partial contamination this signature is sparse. Risk-sensitive retrieval has long treated this situation as the central object of study. The TREC 2014 Web Track Overview introduced $U_{RISK}$, which scores a system by its per-query differences from a baseline and multiplies losses by $(1+\alpha)$. The point was that mean effectiveness hides catastrophic behaviour on a minority of queries. Not All Relevance Scores are Equal ranks documents by Conditional Value at Risk (CVaR), the average of the worst $\beta$ fraction of a score distribution, to penalise fat lower tails. We take over two of these ideas and reverse the direction of the tail. The first is baseline-relative, per-unit deltas. The second is tail averaging. We look for suspiciously fat upper tails of unexplained per-item gains, which is where leaked items sit.

Concretely, let $y_i, y_i' \in \{0,1\}$ be the model's correctness on item $i$ and on its variant. Let $\hat g_i(\theta)$ be the expected original-minus-variant gap for a model of ability $\theta$. We fit $\hat g_i$ with a simple per-item logistic model on a fleet of reference models, and we estimate the target model's $\theta$ from its variant answers only. The excess on item $i$ is $e_i = (y_i - y_i') - \hat g_i(\theta)$, and the audit statistic is

$$T_\beta = \frac{1}{\lceil \beta n \rceil} \sum_{i \in \text{top-}\beta n} e_i ,$$

the CVaR-style mean of the largest $\beta$ fraction of the $n$ excesses. We scan $\beta$ over a small grid and calibrate by resampling reference models to obtain a null distribution. The mean-based statistic is the special case $\beta = 1$. If a fraction $\rho$ of items carries an effect $\Delta$, the mean shifts only by $\rho\Delta$ against noise of order $1/\sqrt{n}$. At $\beta \approx \rho$, the tail statistic isolates effects of size close to $\Delta$. That comparison is the source of the expected power advantage at small $\rho$, and it predicts that the advantage shrinks as $\rho \to 1$.

Relation to Prior Work

The mean-based comparison is ConStat. It fits a hardness-correction function between a benchmark and a reference benchmark across many models, and flags a target whose benchmark score exceeds the prediction by a margin $\delta$. It reports, for example, $\delta \approx 8.2\%$ for Mistral-7b on GSM8K and $\delta \approx 6.6\%$ for Llama-3-70b on ARC-Challenge. ConStat supplies our method's backbone: performance-based detection with reference-model correction. We differ in two ways. We correct at the item level rather than the aggregate level, and we test the tail rather than the mean. This lets us both detect sparse leaks and name the leaked items, which ConStat does not attempt.

On The Fragility of Benchmark Contamination Detection in Reasoning Models constrains the design. It shows that likelihood-based detectors lose most of their signal after reinforcement-learning post-training. LiRA falls from 89.1% AUROC by roughly 14 points after one epoch of GRPO. On models whose contamination happens during reasoning-style fine-tuning, ten detectors sit at 50–58% AUROC despite an 11.7% accuracy inflation. This result rules out any design built on model probabilities, which is why we use only correctness outcomes.

The same paper raises a threat that we take seriously. Such models generalise their confidence to distributionally similar unseen questions. If their accuracy gains also generalise to our variants, the original-versus-variant asymmetry may shrink. We test near variants (rephrasings) against far variants (structural or numeric perturbations) to measure this.

Search-Time Data Contamination motivates the partial regime. It also provides a second testbed with item-level ground truth, because its URL checker labels which items retrieved a mirror. A Narrowing Geometry in Contaminated Reasoning detects contamination from internal activation geometry. It requires model weights and has only been validated on arithmetic. Our audit is black-box and domain-agnostic, and in the white-box setting its eigenspectrum index serves as a complementary check.

On the retrieval side, the TREC finding that robustness is strictly relative to the chosen baseline has a direct analogue here. Our statistic is relative to the reference fleet and to the chosen variant family, so we report sensitivity to both rather than treating the audit as absolute.

Research Program

The work has three parts.

The first part builds paired variant sets and item-level reference models. For GSM8K and the competition-math sets used in the Fragility paper (AIME, AMC, OlympiadBench, Minerva Math), we build numeric and structural perturbations whose answers are checked programmatically. We also build LLM rephrasings, verified by a second model and spot-checked by humans. For MMLU and ARC-Challenge we use rephrasings with the answer options permuted. We evaluate a reference fleet of roughly 20–30 open models of varying ability, matching the scale of ConStat's reference fleet, on originals and variants. From these results we fit the per-item gap model $\hat g_i(\theta)$.

The second part creates controlled contamination with known ground truth. Starting from open base models and from DeepSeek-R1-Distill-Llama-8B, we fine-tune on a fraction $\rho \in \{1, 3, 10, 30, 100\}\%$ of benchmark items, mixed into otherwise clean data. We then optionally apply one epoch of GRPO on clean math data, reproducing the cloaking condition from the Fragility paper. This yields many contaminated and clean model variants with known leaked items.

The third part adds natural search-time contamination. We run a search-augmented open agent on Humanity's Last Exam and SimpleQA, and label items as contaminated using the URL-checker approach of Han et al. We then ask whether $T_\beta$, computed only from the agent's answers on originals and rephrasings, recovers the labelled items without access to retrieval logs. This is a hypothesis, not a given. It holds only if rephrasing reduces the chance that search surfaces the mirror, and we will measure that rate directly.

Baselines

The naive reference point is to report raw accuracy on the original benchmark and do no auditing. Alongside it we report the raw original-minus-variant accuracy gap with no hardness correction. Rephrasings are often slightly harder, so this raw gap will flag clean models too, and it shows how much the correction step contributes.

The strongest existing black-box method is ConStat. We run it with the same reference fleet, the same variant sets and the same bootstrap budget as our method, so the only differences are the ones we are testing. Its reported inflation magnitudes on GSM8K and ARC-Challenge ($\approx 8.2\%$ and $\approx 6.6\%$ in ConStat) come from full-benchmark or large-scale contamination. No reported result was found on Lacuna for ConStat under partial contamination below 10% of items, nor under GRPO cloaking, so we will produce those numbers ourselves. Beating ConStat means strictly higher model-level detection AUROC at $\rho \le 10\%$, at equal false-positive calibration, with no loss at $\rho = 100\%$.

The strongest loss-based methods are LiRA and Min-K%, run as in the Fragility paper. Their reported LiRA AUROC is 89.1% before RL, falling by about 14 points after one GRPO epoch, and Stage II detectors reach 50–58%. We rerun them on our own contaminated models for a like-for-like comparison. Beating them after cloaking means holding AUROC well above their post-RL level.

The most dangerous simple competitor is a per-item paired test with no reference fleet. McNemar's test counts discordant pairs, meaning items where the model is right on one version and wrong on the other. It is sparse-sensitive to some degree and costs nothing to run. We compare against it directly.

Three ablations isolate the proposal's ingredients. The first uses the same item-level correction with $\beta = 1$, which isolates the effect of tail averaging. The second uses tail averaging with only an aggregate hardness correction, which isolates the value of item-level correction. The third is a symmetric version that also treats large negative excesses as evidence, which checks whether looking only at gains is the right choice.

Why not just…?

Why not just run ConStat on the whole benchmark? ConStat measures the mean inflation. At $\rho = 3\%$, even a 30-point per-item effect moves the mean by under one point, which is within the noise of most benchmarks. The $\beta = 1$ ablation is exactly this comparison under identical correction, and the power curve over $\rho$ will show where the two methods diverge. If they do not diverge, the central claim is false, and we will say so.

Why not just use McNemar's test on original–variant pairs? This is a real threat, because it already counts per-item asymmetric flips. Its weakness is that it treats every item as equally prone to flipping. Items differ widely in how much harder their variant is, so genuinely hard-to-rephrase items produce false alarms for strong clean models. If McNemar matches our statistic on clean-model false-positive rates and on localisation, then the reference-fleet machinery is unnecessary, and we would recommend McNemar.

Why not just use fresh or procedurally generated benchmarks such as Humanity's Last Exam or A²RBench? New benchmarks prevent future leaks. They cannot audit claims already made on GSM8K, MMLU or AIME. Search-time contamination also reached Humanity's Last Exam within its release window, so freshness is temporary.

Why not just filter retrieved pages that contain the answer key? For evaluating search agents one does control the retrieval pipeline, and this filter is sensible. It misses paraphrased leaks and discussion pages, however. It also requires retrieval logs, which black-box commercial APIs do not expose. Our audit is complementary: it asks whether leaks happened, from answers alone.

Proposed Experiments and Decisive Checks

The primary metrics are model-level detection AUROC across our contaminated and clean fine-tunes, item-level localisation measured by AUROC and by precision among the top-$\beta n$ items, and the absolute error of estimated $\rho$ against true $\rho$. The first decisive check is the power curve over $\rho$. The prediction is that $T_\beta$ dominates ConStat and McNemar at $\rho \le 10\%$ and converges with them at $\rho = 100\%$.

The second check compares near and far variants after GRPO cloaking. If the contaminated-item asymmetry survives on far variants but vanishes on near ones, performance-based auditing remains viable after RL provided variants are distant enough. If it vanishes on both, then the inflation generalises beyond the leaked items. That would be an important negative result for every performance-based detector, ConStat included.

The third check is external validity on the search agent. We measure agreement between the items $T_\beta$ flags and the items the URL checker labels as contaminated on Humanity's Last Exam and SimpleQA, alongside the measured rate at which rephrasing breaks mirror retrieval.

Finally, we vary the composition of the reference fleet and the variant family, and report how stable the audit's verdicts are. This follows the TREC observation that risk judgements depend on the baseline chosen.