Release-Dated Nulls for Detecting Sparse Benchmark Leakage

Dynamic benchmarks such as LiveBench and LiveCodeBench protect themselves from contamination by adding new questions over time and scoring each model only on questions published after its training cutoff. Contamination here means test questions leaking into a model's training data. The protection rests on trust. Nothing currently checks whether a vendor quietly trained on a few dozen of last month's questions, or whether a model's claimed cutoff is honest. The standard check is to look for a drop in accuracy after the cutoff. That check is weak for two reasons. It averages over all questions, so a leak affecting 5% of a 40-question monthly batch barely moves it. And Test of Time showed that paraphrasing leaked items makes the post-cutoff drop disappear entirely.

We propose a statistical audit that uses time to build a clean reference, not to guarantee cleanliness directly. Questions written after a model's weights were publicly released cannot be in that model's training data. This is a hard guarantee, much stronger than a claimed cutoff. Likewise, an older model released before both the suspect batch and the clean batch is provably clean on both. We use these two facts to build a per-question null distribution that has been corrected for difficulty drift. We then test the suspect batch with the Berk-Jones statistic, a test designed to catch a small hidden fraction of items that behave differently. It does not wait for the average to move. In two sentences: release dates give us guaranteed-clean questions and guaranteed-clean comparison models, which together say how an uncontaminated model should score each suspect question. A sparse-mixture test then asks whether a handful of suspect questions look implausibly familiar, even when the batch mean looks normal.

Core Claim

In the small-batch regime of monthly benchmark releases, fewer than roughly 100 items with a contaminated fraction below 10%, current audits largely fail. Those audits compare a scalar membership score between slices using a mean-type test. We claim that two ingredients, used together, restore usable power while keeping false alarms controlled:

  1. A null distribution certified by release dates and corrected by a clean older model.
  2. An exact tail-sensitive sparse-mixture test on per-item conformal p-values.

Each ingredient does a separate job. The release-dated null removes the difficulty-drift confound that makes raw temporal comparisons untrustworthy. The tail test is the part that gains power when only a few items are leaked.

This is different from the closest work. When Is Benchmark Contamination Detectable? already frames auditing as a sparse mixture $(1-\alpha)P_0+\alpha P_1$ and proves detectability scales with $\alpha\rho\sqrt{m}$. Here $\alpha$ is the leaked fraction, $\rho$ the behavioral separability of seen and unseen items, and $m$ the audit size. It also shows Gaussian power formulas fail in 9 of 9 cases with $m<50$. That paper supplies our problem formulation and constrains our design, since it tells us asymptotic calibration is off-limits. But it compresses per-item scores through mean-type comparisons, works on static injected slices, and assumes a known clean $P_0$. We supply $P_0$ from release dates and replace the mean comparison with a statistic that is optimal for this mixture.

Research Program

Per-item familiarity scores. For each benchmark item $i$ we compute the target model's familiarity score $s_T(i)$. For open-weight models this is the Min-K%++ score from Min-K%++. That method averages the lowest-scoring tokens after normalizing each token's log-probability against the model's vocabulary-wide distribution at that position. For API-only models we use a behavioral score instead: pass rate over repeated samples, combined with the model's confidence on its own solution. We reuse Min-K%++ as the per-item scorer rather than competing with it. Our contribution is what happens after scoring.

Drift correction with a provably clean anchor. We choose one or more anchor models released before both the suspect slice and the clean slice. On the clean slice, which contains items created after the target's release, we fit a monotone regression $g$ that predicts $s_T$ from the anchors' scores. We then form residuals $r(i)=s_T(i)-g(s_A(i))$, where $s_A$ is the anchor score. Under no contamination, residuals on suspect and clean items are exchangeable, provided the suspect-versus-clean difficulty shift affects target and anchor alike.

This resembles the reference-model curve in ConStat, which fits expected performance from a panel of reference models on rephrased benchmarks. There is one essential difference. ConStat must trust that its reference models are clean, and it breaks if a whole model generation saw the benchmark. Our anchors are clean by construction.

Conformal p-values. We split the clean slice into a fitting half and a calibration half of size $n$. For each suspect item we compute a one-sided conformal p-value: essentially the rank of its residual among the calibration residuals.

$$p_i=\frac{1+\#\{j:\ r(j)\ge r(i)\}}{n+1}.$$

A small $p_i$ means item $i$ is unusually familiar to the target, relative to what a clean model predicts.

Berk-Jones test. We apply the one-sided Berk-Jones statistic $M_n^+=\sup_t K(F_m(t)/t)$ to the $m$ suspect p-values. Here $F_m$ is their empirical distribution function and $K(x)=x\log x-x+1$. On the exact Berk-Jones statistics proves this statistic is adaptively optimal for rare-and-weak mixtures without knowing the leaked fraction. That paper also supplies an exact, numerically stable p-value algorithm, which is the method we reuse.

One technical problem is new. Conformal p-values that share a calibration set are exchangeable but not independent, while the exact algorithm assumes independent uniforms. We therefore calibrate the test by permuting pooled residuals, which is exactly valid under exchangeability. We then measure how far the fast exact algorithm drifts from this permutation result. We also report an estimate of the leaked fraction taken from where the supremum is attained. This gives vendors and benchmark maintainers an interpretable quantity, not only a rejection.

Stress test against reinforcement-learning cloaking. On The Fragility of Benchmark Contamination Detection shows that a clean GRPO phase after contaminated fine-tuning collapses likelihood detectors toward chance. GRPO is a reinforcement-learning method for post-training language models. The paper attributes the collapse to clipping that compresses the variance of non-member losses. That is a change in shape, not location. The collapse is therefore a direct test of whether a tail statistic recovers signal that mean detectors lose. This result could just as easily be negative, and we treat it as an open question, not a selling point.

Baselines

The naive reference point is to do nothing: accept the release-date partition, as LiveBench and LiveCodeBench do, and declare post-cutoff scores clean. Its implicit false-negative rate is 100% by construction. We also run a naive active check: a two-sample t-test on pass rate or Min-K%++ score between the suspect and clean slices, with no drift correction. This is the formal version of the post-cutoff accuracy cliff that LiveCodeBench uses. We expect it to have inflated false alarms under difficulty drift and low power at small leaked fractions.

The strongest existing method is the power-calibrated audit from When Is Benchmark Contamination Detectable?. It uses a scalar membership score, a two-sample comparison, and a simulation-based two-stage sample-size planner. It reports achieved power of 0.785–0.92 against a 0.80 target on its SQuAD-in-the-Pile injection set. We run it on exactly our injected checkpoints, with the same Min-K%++ scorer, the same clean slice, the same audit sizes, and the same 0.05 level. We give its planner the same simulation budget we use for permutation calibration. Beating it means our method reaches power 0.8 at a leaked fraction at least a factor of two smaller for a fixed $m\le 60$, while holding empirical false-alarm rate at or below 0.05 on clean-versus-clean splits.

We also report Min-K%++ as a standalone per-item detector by AUROC, for continuity with Min-K%++. That paper reports a 6.2 to 10.5 point AUROC gain over Min-K% on WikiMIA. Finally, we run ConStat under its published protocol on the same items. Its reported 98.4% detection of sample-specific contamination comes from larger static benchmarks. No reported result on Lacuna covers ConStat or Berk-Jones on small monthly slices.

Three ablations isolate the ingredients. The first keeps Berk-Jones but drops the anchor correction, so the null is the raw clean slice. It measures how much the drift correction protects the false-alarm rate. The second keeps the anchor correction but replaces Berk-Jones with a Mann-Whitney test on the p-values. It measures what the tail statistic adds. The third replaces Berk-Jones with a two-sample Anderson-Darling test, which is tail-weighted but not optimal for the mixture. All variants share scorer, splits, and permutation calibration.

Why not just…?

Why not just check for a post-cutoff accuracy drop? This check has already been shown to fail in both directions. Test of Time found that turning arXiv papers into reasoning questions erases the drop. Models even scored 2.19 points higher on post-cutoff physics. Renaming variables on LiveCodeBench also removes the cliff. So a missing drop certifies nothing, and a present drop may reflect difficulty drift. Our no-anchor ablation and the t-test baseline measure the drift problem directly on clean-versus-clean splits taken from different months.

Why not just run Min-K%++ and threshold it per item? Per-item membership AUROC is not the audit question. An auditor needs a calibrated batch-level decision at small $m$. Averaging or thresholding per-item scores throws away the information that only a few items are anomalous. Min-K%++ remains our scorer, so the comparison isolates the aggregation step. If the mean-based audit with Min-K%++ matches our power at $\alpha\le 0.1$, the tail-statistic claim fails, and we will say so.

Why not just use Anderson-Darling or Kolmogorov-Smirnov, which are simpler? This is a real threat. On the Empirical Power of Goodness-of-Fit Tests in Watermark Detection found classical goodness-of-fit tests already beat scalar detectors by wide margins. Kolmogorov-Smirnov loses sensitivity at the edges of the distribution, where all distribution functions converge to 0 and 1. Anderson-Darling up-weights the tails and may come close to Berk-Jones. The ablation settles this. If Anderson-Darling matches Berk-Jones, the contribution reduces to the release-dated, drift-corrected conformal null, and we would recommend Anderson-Darling for its simplicity. Theory predicts Berk-Jones should pull ahead mainly at the smallest leaked fractions.

Why not just use ConStat? ConStat's reference panel must itself be clean. That assumption becomes shakier each year as shared web corpora absorb benchmarks. It also needs many rephrased prompts and many models. We replace trusted references with anchors that are clean by release date, at the cost of needing post-release clean items. That cost limits us to auditing models some weeks after release.

Proposed Experiments

Controlled injection. We continue pretraining an open-weight base model on an open corpus, following the OLMo2 setting used in Test of Time. We inject a fraction $\alpha\in\{0.02,0.05,0.1,0.2\}$ of a LiveCodeBench or LiveBench monthly slice. Injection is either verbatim or paraphrased, matching the exposure hierarchy (verbatim, then paraphrase, then surface perturbation, then answer-only) documented in When Is Benchmark Contamination Detectable?. Clean items come from later releases of the same benchmark. Anchors are earlier public checkpoints. We sweep $m\in\{30,60,120\}$.

The primary metric is power at level 0.05, together with the leaked fraction needed to reach power 0.8. The decisive false-alarm check runs every method on pairs of clean slices from different months, measuring empirical type I error under real drift, including the post-2024 "ChatGPT effect" on arXiv-style text documented in Is Your LLM Outdated?. As a scorer sanity check we reproduce WikiMIA, the Wikipedia benchmark that splits articles by date, to confirm Min-K%++ behaves as reported.

Reinforcement-learning cloaking. We reproduce the fine-tune-then-GRPO pipeline of On The Fragility of Benchmark Contamination Detection on a small model with AIME, AMC and OlympiadBench items. That paper reports LiRA falling from 89.1% AUROC toward chance. We ask whether batch-level Berk-Jones power survives where the per-item mean detectors do not.

Real-world audit. We apply the finished test to open-weight models on LiveCodeBench time bins and on LiveBench monthly releases, and report estimated leaked fractions with confidence intervals. Any flag is a hypothesis to examine against open training data where available, not an accusation.

The project succeeds if three things hold together:

  1. False-alarm rate stays at or below 0.05 across drifting clean-versus-clean splits.
  2. The detectable leaked fraction at $m\le 60$ is at least halved relative to the power-calibrated scalar audit.
  3. The Mann-Whitney and Anderson-Darling ablations show which ingredient carries the gain.