Counting the Extremes: Tail-Calibrated Contamination Audits for Reasoning Models

When a language model has seen a benchmark's test questions during training, its leaderboard score partly reflects memory rather than skill. Current detectors ask whether a model's confidence on benchmark items differs on average from its confidence on items it has not seen. That question is becoming unanswerable. After reinforcement-learning post-training, or after fine-tuning on long step-by-step solutions, the average gap between seen and unseen items shrinks until detectors perform near chance. We propose asking a narrower question: are there more benchmark items in the extreme tail of the detector score than a clean model would produce? We would answer it with extreme value theory, the branch of statistics that models the far tail of a distribution. With that tool, the false-alarm rate can be set at levels far below what a small clean reference set can measure directly, and the audit can report a statistically guaranteed lower bound on how many items were memorized.

The idea in two sentences: calibrate the far upper tail of a contamination score using clean reference items and a fitted tail model, then count how many benchmark items exceed an extreme cutoff. Too many exceedances means contamination, even when the bulk of the score distribution looks clean. The part of extreme value theory that does the work is threshold selection, meaning deciding where "the tail" begins. The application regime is the post-training setting where mean-based detectors fail.

Core Claim

On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that one epoch of GRPO lowers LiRA's AUROC by about 14% from 89.1% and pushes reference-free detectors toward 50%. After contaminating distilled reasoning models, ten detectors average 50–58% AUROC while pass@1 is inflated by 11.7%. AUROC is a whole-distribution ranking statistic. It can sit near 0.5 while a minority of contaminated items still receive scores that clean items essentially never reach. Our claim is conditional and testable. If memorization survives post-training for even a fraction of items as extreme scores, then an exceedance test calibrated by a Generalized Pareto tail model will detect contamination with controlled false-alarm rates in regimes where paired $t$-tests, rank tests and AUROC-style detectors fail. If no such tail signal survives, the experiments will show that too. That outcome would itself be informative, because it would say the cloaking reaches the extremes and not just the bulk.

What would be new is not "EVT meets memorization." PRIVET already models nearest-neighbor distances between generated and training samples as extreme values in order to audit image, genomic and tabular generative models. It neither addresses likelihood-based scores of autoregressive models nor treats threshold choice as a problem. The novelty here lies in three things together: a dataset-level exceedance test built for benchmark auditing, principled selection of where the tail starts, and a finite-sample lower bound on the number of contaminated items.

Method

For each benchmark item $i$ we compute a paired score $d_i = s(x_i) - s(\tilde{x}_i)$. Here $s$ is a per-item detector score (MIN-K% PROB from Detecting Pretraining Data from Large Language Models, sequence loss, or PaCoST's $P(\text{True})$ confidence), $x_i$ is the canonical item and $\tilde{x}_i$ is a meaning-preserving rephrase. The pairing follows PaCoST. It cancels most item difficulty, which is the main reason a clean reference set and a target benchmark would otherwise differ. We compute the same paired score on a clean reference set: items released after the model's training cutoff, or in controlled experiments, held-out items from the same benchmark.

On the reference scores we fit the peaks-over-threshold model. Scores above a threshold $u$ have their excesses fitted to a Generalized Pareto distribution with scale $\sigma$ and shape $\xi$. The cutoff exceeded by a clean item with probability $q$ then follows the SPOT formula of Anomaly Detection in Streams with Extreme Value Theory:

$$z_q = u + \frac{\sigma}{\xi}\left(\left(\frac{q\,n_{\text{ref}}}{N_u}\right)^{-\xi} - 1\right),$$

where $n_{\text{ref}}$ is the reference size and $N_u$ is the number of reference scores above $u$. The formula remains usable for $q \ll 1/n_{\text{ref}}$, which is exactly where empirical quantiles and conformal ranks run out of resolution.

Let $N$ be the number of the $n$ benchmark items with $d_i > z_q$. Under the null of no contamination, $N$ is approximately Binomial$(n, q)$. We propagate the uncertainty in the tail fit through an upper confidence bound $\bar{q}$, which gives a test and a count bound:

$$m_{\text{low}} = N - \text{Binom}^{-1}_{1-\alpha}(n, \bar{q}).$$

We report $m_{\text{low}}$ as "at least $m_{\text{low}}$ items are contaminated at confidence $1-\alpha$." This is the item-count analogue of ConStat's score-inflation estimate.

Threshold selection is the step where the extreme value literature does real work. If $u$ is too low, bulk scores bias $\xi$. If it is too high, $z_q$ becomes very unstable. We select $u$ by minimizing a bootstrap estimate of the mean squared error of $z_q$ over a grid of candidate thresholds. This follows Regenerative block-bootstrap confidence intervals for tail and extremal indexes, which uses the bootstrap to locate the bias-variance minimizing threshold. Because reference items are independent, an ordinary bootstrap suffices, and we do not need their regeneration machinery for Markov chains. We check stability of the selected $u$ against the parameter-stability criterion used by Kiriliouk et al.. The interval on $z_q$ comes from the profile likelihood rather than the delta-method Wald interval. Haeusler and Segers show that Wald intervals for tail quantities can cover only 85% at nominal 95% in finite samples, while likelihood-ratio intervals with Bartlett correction are far more accurate. Their result concerns the Hill estimator, so it motivates our choice without guaranteeing it for the Generalized Pareto fit. For that reason we check coverage by simulation.

Research Program

The work has three parts. The first is a measurement study: does a tail signal survive post-training? We reproduce the two-stage contamination protocol of the Fragility paper. In the first stage, we fine-tune on a known subset of items and then apply GRPO. In the second, we contaminate DeepSeek-R1-Distill-Llama-8B with chain-of-thought solutions. Before and after each stage we record the full distribution of $d_i$ for member and non-member items. We report AUROC, but mainly the true-positive rate at false-positive rates of 1%, 0.1% and 0.01%. We also estimate the tail shape $\xi$ separately for members and non-members. That paper argues that GRPO's clipping shrinks the variance of non-member losses. We test directly whether that shrinkage also compresses the upper tail of member scores, or leaves an exceedance gap.

The second part is to build the audit and establish its calibration. That means the threshold-selection procedure, the interval on $z_q$, the exceedance test, and the bound $m_{\text{low}}$. We verify false-alarm rates on clean models and clean splits across many reference and target resamplings.

The third part is to apply the audit. We run it on the benchmarks where PaCoST and ConStat reported contamination (ARC, GSM8K, MMLU, HellaSwag) and on WMDP, which PaCoST treats as a clean control. Where our verdicts disagree with theirs, we examine the disagreeing items by hand.

Baselines

The naive reference point is a fixed-percentile exceedance test. It counts target items above the empirical 95th or 99th percentile of reference scores and compares the count to a binomial. It needs no tail model, and it is the right "do nothing" comparison because it shares our test's logic and differs only in how the cutoff is set. We run it on the same scores and reference sets. Beating it means higher power at matched, verified false-alarm rates once the target rate falls below $1/n_{\text{ref}}$. Above that rate we expect to tie with it, and we will say so.

The strongest existing dataset-level method is PaCoST's paired $t$-test on $P(\text{True})$ at $p<0.05$ (PaCoST). It shares our pairing, so any difference isolates mean-based versus tail-based testing. We also run a Wilcoxon signed-rank variant, which removes the normality objection, and ConStat (ConStat) with the same reference models on GSM8K and ARC. ConStat reports detecting 98.4% of sample-specific contamination against 71% for the best prior baseline, in its own controlled setting. That figure is from settings without RL post-training. We found no reported result on Lacuna for PaCoST or ConStat after GRPO or on the Stage II reasoning-model protocol, so we produce those numbers ourselves. For per-item detection the reference numbers are LiRA's 89.1% AUROC before GRPO and the 50–58% average AUROC of ten detectors in Stage II, both from the Fragility paper. We found no reported result on Lacuna for true-positive rate at low false-positive rate in that setting. Beating the baselines means rejecting the null at matched false-alarm rate, and with a tighter nonzero $m_{\text{low}}$, on contaminated post-GRPO models where PaCoST, Wilcoxon and ConStat do not reject. All methods receive the same scores, the same rephrases and the same reference items. Any tuning uses only reference data.

Two ablations isolate the key ingredient. The first replaces bootstrap threshold selection with fixed thresholds at the 90th and 97th reference percentiles; this tests whether choosing where the tail starts matters. The second replaces the profile-likelihood interval with a Wald interval; this tests whether interval construction drives calibration.

Why not just…?

Why not just use an empirical quantile or a conformal p-value from the reference set? Doing so is valid and assumption-light. It is a real threat whenever the reference set is large relative to $1/q$. Its limit is resolution. With a few hundred post-cutoff rephrased items, the smallest achievable false-alarm rate is roughly one over the reference size, and competition benchmarks such as AIME and AMC are tiny. Contamination that affects only a handful of items is detectable only at cutoffs rarer than that, where an empirical cutoff has no data behind it. The fixed-percentile baseline and a conformal variant are both in the experiments. If they match our power at every false-alarm rate they can reach, and our advantage appears only at rates we cannot verify, we will report that honestly. For that reason we verify extreme false-alarm rates on large pooled clean splits that are held out from the fit.

Why not just use a rank test or the paired t-test that PaCoST already runs? Both tests measure shifts in the location of the bulk of the distribution. The Fragility paper's AUROC near 50% implies that a rank-sum statistic has almost no signal, because AUROC is the normalized Mann–Whitney statistic. If contamination survives only in a minority of items, a mean or rank test dilutes it over the whole benchmark. The measurement study decides the question. If member and non-member tails coincide after GRPO, the exceedance test will be no more powerful than these tests, and the proposal's power claim fails.

Why not just tune k in MIN-K% or the score threshold on a labeled validation set? In an external audit there are no labeled contaminated items; that is the whole difficulty. A tuned threshold also has no stated false-alarm guarantee. MIN-K%'s own gain, 7.4% AUC on WikiMIA (Detecting Pretraining Data from Large Language Models), came from discarding the bulk of tokens, which is the same intuition we apply one level up, to items. We deliberately do not propose fitting a separate tail model per sequence to choose $k$. A benchmark item contributes only tens of tail tokens, and a per-sequence Generalized Pareto fit on that few is too unstable to beat a well-tuned global $k$.

Proposed Experiments and Decisive Checks

The first decisive check is calibration. On uncontaminated base models, and on clean held-out splits of AIME, AMC, GPQA Diamond, OlympiadBench, Minerva Math, GSM8K and WMDP, the realized false-alarm rate of the exceedance test must stay within Monte Carlo error of nominal at $\alpha = 0.05$ with $q$ down to $10^{-3}$. We verify this against large pooled clean data held out from the tail fit. Miscalibration caused by mismatch between reference and target items would appear here first. If it appears, we report it as the method's limit.

The second is survival of the tail signal. We compare true-positive rate at 0.1% false-positive rate for members versus non-members before and after one and several GRPO epochs, and in Stage II. The proposal predicts that this rate degrades much less than AUROC does.

The third is power. We vary the contaminated fraction from 5% to 50% of a benchmark and plot rejection rates for our test, the fixed-percentile test, the Wilcoxon test, PaCoST and ConStat, and we report $m_{\text{low}}$ against the true member count.

The fourth is the two ablations, which show whether threshold selection and interval construction actually drive the result.

On real public models, the result we are after is agreement with ConStat and PaCoST where they reported strong contamination, such as Mistral-7b-v0.1 on GSM8K, and nonzero lower bounds on models that mean-based tests call clean. Each such bound can be examined item by item.