Forecasting the Sensitivity of Contamination Audits Before Running Them

When someone checks whether a language model saw a benchmark during training, the result usually comes back as "contaminated" or "no evidence found." The second answer is close to uninformative. It could mean the benchmark is clean, or it could mean the test was too weak to notice a few hundred leaked items. Physicists face the same problem and handle it routinely. Before building a telescope or a detector, they write down a model of the signal and of the noise. From it they compute the Fisher information, a number that says how sharply the data can pin down an unknown quantity. Inverting it gives the Cramér-Rao bound, the smallest error any unbiased estimator can reach. That bound tells them in advance what the experiment can resolve. After a non-detection, it lets them publish an upper limit instead of a shrug.

We propose importing this practice into contamination auditing. The unknown quantity is the fraction of a benchmark that leaked into training, together with how strongly it was memorized. The "instrument" is a detection statistic, such as order-likelihood gaps, perturbation sharpness, or paraphrase gaps. A small set of controlled injection runs measures the signal and noise of each detector. Fisher forecasting then predicts, for any benchmark size and any combination of detectors, the minimum contamination level that can be detected. That prediction does three things. It turns published non-detections into quantitative upper limits. It tells auditors which detectors to combine and how many items they need. It shows which contamination types can be told apart and which ones look the same to every available detector.

Core Claim

Contamination detectors already exist and some are strong. What the field lacks is a sensitivity model for them. Every method in the file reports power only post hoc, on its own test conditions, so nobody can say what a given audit could have caught. The Sharded Rank Comparison Test asks whether a model prefers the benchmark's canonical item order over shuffled orders. It is the most rigorous black-box test in the file. Its authors show it detects datasets seen 2 to 4 times in a 1.4B-parameter model trained from scratch. They also report "little evidence" of contamination on GSM8K for LLaMA2 and Mistral. Meanwhile PAC, which measures how sharply token confidence changes under word swaps, reports a 91.4% contamination rate on GSM8K for davinci-002. The two results concern different models, so they do not contradict each other. Still, neither paper can say what level of GSM8K contamination its own null result rules out. We claim that a calibrated Fisher forecast can answer exactly that question, and that it will predict empirically measured detection power to within a small factor across benchmarks, repetition counts and contamination types.

The core calculation is short. Let $s_i$ be a detector's statistic on item (or shard) $i$ of a benchmark with $n$ items. Let $f_0$ be its distribution on clean items, and $f_1$ its distribution on items that were seen $k$ times at a given contamination level. Let $\pi$ be the unknown leaked fraction. The data follow the mixture $(1-\pi)f_0 + \pi f_1$. The Fisher information about $\pi$ near the clean hypothesis $\pi = 0$ is

$$I(0) = n \, \chi^2(f_1 \,\|\, f_0), \qquad \chi^2(f_1\|f_0) = \mathbb{E}_{f_0}\!\left[(f_1/f_0 - 1)^2\right],$$

so the minimum detectable fraction at significance multiplier $z$ is approximately

$$\pi_{\min} \approx \frac{z}{\sqrt{n\,\chi^2(f_1\|f_0)}}.$$

This follows from the Cramér-Rao bound, since the standard error of $\hat\pi$ is at least $I(0)^{-1/2}$. For a Gaussian shift of size $\Delta(k)$ against noise $\sigma$, the divergence $\chi^2$ equals $e^{\Delta(k)^2/\sigma^2} - 1$. Detectability therefore depends on a signal-to-noise ratio that grows with repetition $k$ and shrinks under paraphrase.

Several detectors can be stacked into one vector statistic. The joint Fisher matrix then covers both $\pi$ and the contamination level (verbatim question-answer pairs, questions only, or paraphrases, following the levels in the contamination survey). This plays the same role as combining galaxy surveys with background-radiation data in Measuring Cosmological Parameters with Galaxy Surveys. Each detector alone cannot separate "a little verbatim leakage" from "a lot of paraphrased leakage," but detectors that respond differently to the two can break that ambiguity.

Relation to Prior Work

The Fisher machinery itself is borrowed from the physical sciences. The cosmology paper above, the evidence-forecasting work in Forecasting the Bayes factor of a future observation, and the localization bounds in Cramér-Rao Bound of Joint DOA-Range Estimation for Coprime Frequency Diverse Arrays all forecast what an experiment can resolve from a signal-and-noise model. We reuse that logic unchanged. The new parts are the "instrument," which is a contamination detector, and the parameter, which is the leaked fraction.

Within machine learning evaluation, the closest use of Fisher design is Fewer Battles, More Gain. It chooses Chatbot Arena matchups to shrink the variance of model ratings in the Bradley-Terry model, a statistical model of pairwise wins. It assumes clean evaluation and estimates skill, not leakage. We use the same design criteria, but on detector choice and audit size.

Several papers put Fisher geometry inside network weights. The Geometry of Forgetting uses the Fisher matrix of an alignment dataset to predict how much alignment is lost when weights change. Unlearning Isn't Deletion uses it as a diagnostic during unlearning. Fisher-Rao Sensitivity for OoD detection scores vision inputs by the trace of the Fisher matrix and reports 90.61% AUROC on ImageNet versus iNaturalist. Test of Time uses a Kronecker-factored Fisher approximation (EK-FAC) to compute influence functions, which trace a model's answers back to specific training documents. All of these need white-box access, and they use the Fisher matrix as a detector or an editing tool. Our Fisher matrix lives in the much smaller space of the audit's own parameters. That makes the approach cheap and usable on black-box models. It also means we do not claim weight-space curvature reveals contamination.

The detection literature supplies the instruments: the sharded rank test, PAC, and token-level membership inference via InfoRMIA. Test of Time constrains the design. It shows that paraphrasing drops influence hit rates from 77.5% to 17.5%, and that perturbing LiveCodeBench problems erases the drop in accuracy on problems published after the training cutoff. In our terms, paraphrase shrinks $\Delta$ for some detectors. A forecast must therefore model the contamination level explicitly rather than assume verbatim leakage.

Research Program

The first component is a calibration testbed. We would reproduce the controlled-contamination setup of the sharded rank paper. Small GPT-2/Pythia-scale models are pretrained on a FineWeb subset, with BoolQ, HellaSwag, MMLU, GSM8K and AI2-ARC items injected. Injections vary repetition $k$ over $\{1,2,4,8,16\}$, the leaked fraction $\pi$, and the contamination level: verbatim question-answer pairs, questions only, or LLM paraphrases in the style of Test of Time. On each model we run four detectors. They are the sharded rank statistic, PAC's polarized distance, a minimum-token-probability membership score, and a paraphrase gap, meaning the accuracy drop from original to rewritten items.

The second component is the forward model. From a subset of runs we fit each detector's clean distribution $f_0$, using held-out never-injected items and post-cutoff items. We also fit the contaminated response $f_1(k, \text{level})$ and the correlations between detectors. These feed the joint Fisher matrix, so we can read off minimum detectable fractions and the ambiguities between parameters.

Calibration on small models will not transfer unchanged to frontier models. So the third component re-estimates the noise model per target model from items that are clean by construction, such as freshly generated variants and post-cutoff problems. It then treats the signal response $\Delta(k)$ as the main uncertainty and propagates it into a band on $\pi_{\min}$.

The final product is an audit protocol. For each target model and benchmark, it either reports an estimate of $\pi$ with its error, or reports an upper limit. It also gives the optimal detector combination and item budget, chosen to minimize the forecast variance of $\hat\pi$ under a fixed query budget.

Baselines

The naive reference point is current practice. Run one detector, apply its significance threshold, and read a non-significant result as "clean." Against this baseline we measure how often a non-detection coincides with injected contamination well above zero. This is the failure the proposal is meant to expose. Beating it means our upper limits contain the true $\pi$ at the stated rate. Under the naive reading, the implied claim of "zero contamination" is false in those runs.

The strongest existing detectors are the sharded rank test and PAC, run exactly as published on identical models and items. The sharded rank paper reports reliable detection at 2 to 4 duplicates and p-values as low as $10^{-38}$ for datasets seen 10 or more times, with test sets of 1000 examples. PAC reports detection AUC gains of 4.5% on WikiMIA and 5.9% on StackMIA over the best prior baselines. Both serve two roles. They are instruments inside our forecast, and they are competitors for actual detection. For detection, the Fisher-optimal combined test beats them if it reaches a higher true positive rate at 1% false positive rate on the injected testbed at equal query budget. For forecasting, they have no forecast of their own. The comparison there is to the next baseline.

The strongest competitor for the forecasting claim is brute-force empirical power analysis. For each configuration, one retrains or reuses injected models, runs the detector many times, and records the detection rate. This is the ground truth we forecast against. It is also the honest alternative method when compute allows. We give it the same injection runs. The forecast wins if it predicts held-out empirical $\pi_{\min}$ within a factor of 1.5 for configurations it was not fitted on, at a small fraction of the compute.

Two ablations isolate the key ingredients. The first replaces the joint Fisher matrix with separate per-detector forecasts that ignore correlations. This tests whether combining detectors to break ambiguities matters. The second replaces the analytic Gaussian-shift form with a parametric bootstrap from fitted densities. This tests whether the Cramér-Rao approximation itself is adequate or merely the modeling.

Why not just…?

Why not just run a simulation-based power analysis instead of a Fisher forecast? This is the real threat. If the fitted forward model is good, simulating from it gives the same answer as the analytic bound, and simulation handles heavy tails better. Our position is that the expensive part of either approach is calibrating $f_0$ and $f_1$, and we share that cost. On top of it, the Fisher form adds two things. It gives closed-form scaling in $n$ and $k$, which lets the forecast extrapolate to benchmark sizes not tested. It also gives the ambiguity structure across contamination levels. The bootstrap ablation settles the question directly. If the bootstrap forecasts empirical power clearly better, we will report the protocol as "calibrate, then simulate" and keep the Fisher matrix only for design and ambiguity analysis.

Why not just report the p-values existing tests already produce? A p-value measures surprise under the clean hypothesis. It says nothing about which contamination levels the test would have missed. The GSM8K "little evidence" result in the sharded rank paper is exactly such a case. Converting nulls into upper limits requires a model of the contaminated alternative, and that is what we build. Experiment two below tests whether the resulting limits are honest.

Why not just use bigger benchmarks or fresh, dynamic ones like LiveBench? Larger $n$ helps only as $1/\sqrt{n}$, and fixed benchmarks cannot grow. LiveCodeBench, for example, has 612 problems, so knowing the per-item signal-to-noise ratio is what tells you whether size suffices. Fresh benchmarks such as LiveBench and LiveCodeBench reduce contamination but do not measure it. Test of Time shows their main check, the accuracy drop after the training cutoff, can be erased by rewriting. We instead use their post-cutoff items as the clean reference that calibrates $f_0$.

Proposed Experiments

The first decisive check is forecast accuracy. We fit forward models on injection runs at $k \in \{1, 4, 16\}$ and verbatim leakage. We then predict empirical minimum detectable fractions at $k \in \{2, 8\}$, at paraphrased leakage, and at benchmark sizes from 100 to 2000 items. The forecast is useful if predicted and measured $\pi_{\min}$ agree within a factor of 1.5. It fails if the error grows without bound under paraphrase, which is the regime where Test of Time suggests signals collapse.

The second check is upper-limit coverage. On injected models where the detectors return non-significant results, we check that the 95% upper limit on $\pi$ contains the true injected fraction at least 95% of the time.

The third check is ambiguity breaking. We test whether the joint forecast correctly predicts which detector pairs can separate verbatim from paraphrased leakage, by comparing the predicted and measured errors in recovering the contamination level.

The fourth experiment is an audit of open models already examined in prior work: LLaMA2-7B, Mistral-7B and Pythia-1.4B on MMLU, AI2-ARC and GSM8K. Here we do not know the true contamination, so the forecast cannot be verified directly. The aim is to express prior findings in our terms. The sharded rank paper reports p ≈ 0.01 on MMLU and p = 0.001 for Mistral on AI2-ARC. We restate these as estimates or upper limits on contamination fraction and repetition, and we report what audit size and detector combination would be needed to settle the GSM8K question. The fair test of transfer is the per-model noise calibration on clean-by-construction items. If transferred forecasts systematically disagree with what detectors actually find on these models, that disagreement is itself the finding: small-model calibration of $\Delta(k)$ does not carry over to larger models.