Relearning Speed as a Detector of Dormant Benchmark Contamination

A language model that saw a benchmark's test questions early in pre-training can look clean by the end of it. Continued training on ordinary web text pushes the leaked items into a "dormant" state: the score gap between contaminated and clean items nearly disappears, and perplexity-based audits find nothing. Ordinary fine-tuning then brings the memorized answers back and inflates scores by up to 4% (The Impact of Post-training on Data Contamination). This proposal tests whether the leak can be caught before that happens. The method gives the model a short burst of fine-tuning on the benchmark items. Items it has seen before should regain low loss much faster than items it has not, in the same way people relearn a forgotten language faster than they learn a new one.

The tool for reading that signal comes from the noisy-label literature. There, dynamic loss thresholding (DLT) means separating good training examples from corrupted ones by watching how each example's loss changes during training. The cutoff is set from the current loss distribution rather than fixed in advance. Learning From Noisy Labels via Dynamic Loss Thresholding observed that clean samples show a sharp loss drop between early and late training while noisy samples do not. It fit a two-component Gaussian mixture to that drop and recovered the noise fraction almost exactly, estimating 59.1% on a 60%-noise dataset. We apply the same machinery to a deliberately induced, short training trajectory. The goal is to classify individual benchmark items as previously seen or unseen and to estimate what fraction of a benchmark is dormant-contaminated.

Core Claim

Dormant contamination leaves no mark on a model's loss at a single checkpoint, but it should leave a mark on the loss trajectory under brief re-exposure. We define an item's relearning score from $k$ steps of probe fine-tuning on a disposable copy of the model:

$$s_i = \frac{\ell_i(0) - \ell_i(k)}{\ell_i(0)} - \frac{\ell_{\tilde i}(0) - \ell_{\tilde i}(k)}{\ell_{\tilde i}(0)}.$$

Here $\ell_i(t)$ is the answer-token loss on item $i$ after $t$ probe steps, and $\tilde i$ is a matched control: a meaning-preserving paraphrase of item $i$ with a freshly written solution, put through the same probe. Subtracting the control's relative drop removes the obvious confound that easy items relearn fast whether or not they leaked. We then set the decision threshold the DLT way. We fit a two-component mixture to the distribution of $s_i$ across the benchmark, read off the contaminated component's weight as the estimated contamination fraction, and use the posterior as the per-item flag. No absolute loss cutoff is chosen by hand.

What is new is the combination: the idea of a self-calibrating cutoff on loss change (not loss level), from DLT, applied to the specific failure that The Impact of Post-training on Data Contamination documented but did not try to detect.

How This Relates to Prior Work

Three lines of contamination detection read a model at one moment. Estimating Contamination via Perplexity compares benchmark perplexity to "memorized" and "clean" reference corpora. ConStat checks whether accuracy carries over to rephrased reference items. RADAR trains a classifier on layer-by-layer confidence and attention features from a single forward pass, using only 30 training examples. All three assume that memorization is visible in the current weights' behavior. The dormant-contamination result is direct evidence against that assumption after long clean pre-training, and this is the gap our proposal targets. We borrow ConStat's use of rephrasings, but as a calibration control for a dynamic signal rather than as the signal itself.

Detecting Data Contamination from Reinforcement Learning Post-training motivates the move away from static likelihood. It shows that perplexity and Min-K% (the average log-probability of a sequence's least likely tokens) fall to 0.42–0.53 AUC when contamination entered through reinforcement learning. Its Self-Critique probe is still an inference-time entropy measurement and is restricted to verifiable reasoning tasks. Our probe needs no answer checker, though it does need weight access.

The noisy-label papers constrain the design as much as they supply it. Weed Out, Then Harvest argues that fixed loss thresholds fail because the right cutoff shifts across samples and training stages, which is why we use a mixture-fit cutoff. AlleNoise shows models memorizing up to 70% of semantically plausible wrong labels. That is a warning that "plausible" items can be absorbed quickly whether or not they leaked. The paraphrase control exists to absorb exactly that effect. Finally, Investigating Data Contamination for Pre-training Language Models supplies the experimental template: GPT-2 pre-training with injected benchmark text at controlled repetition counts. It also supplies the finding that the effect of leaked data is non-monotone in repetition count, which our probe should reproduce if it measures the right thing.

Research Program

The core work is a controlled study in which ground truth is known. We pre-train small models on FineWeb and inject a random half of GSM8K and MBPP test items into the early token stream, following the protocol of The Impact of Post-training on Data Contamination: 5 copies within the first 2B of 25B tokens. The non-injected half serves as within-model clean items. A twin model trained without injection serves as a between-model control. We vary three factors. The first is contamination level: question-only versus question-plus-answer, matching the data-level and label-level contamination categories of DCR. The second is repetition count, from 1 to 20 as in the GPT-2 study. The third is how many clean tokens follow the injection, which controls how dormant the leak is. We also inject an MMLU subset so the method is tested on a knowledge benchmark, not only on reasoning.

For each checkpoint we run the probe with LoRA adapters, which are small low-rank weight updates that leave the base model untouched. Probe settings are $k \in \{1, 4, 16\}$ steps and a learning rate swept once on a held-out injected benchmark. Paraphrase controls are generated by a separate model and filtered for answer equivalence. We record the full per-step loss curve, not only its endpoints, and test whether the curve's shape adds information beyond $s_i$.

The analysis then asks whether relearning score predicts contamination when static perplexity cannot. It also asks whether the mixture-estimated contamination fraction tracks the true 50%, and whether an item's score predicts how much that item gains under subsequent SFT. That last question is the practical one: can the probe forecast resurfacing before it happens? Finally, we apply the calibrated probe to open models with suspected leaks. ConStat reports a GSM8K contamination gap of $\delta \approx 8.3\%$ for Mistral-7b-v0.1 and an ARC gap of $\delta \approx 6.6\%$ for Llama-3-70b. We check whether our estimated contamination fractions are non-trivial on those pairs and near zero on post-cutoff controls.

Baselines

The naive reference point is static perplexity and Min-K% scored on the final pre-training checkpoint. Items below a mixture-fit cutoff are flagged, so that the thresholding rule is identical to ours and only the signal differs. This is the standard detector and the one the dormant-contamination result predicts should fail. The file has no reported AUC for it in the dormant pre-training setting. The closest reported figure is 0.42–0.53 AUC for PPL and Min-K% under RL-stage contamination (Detecting Data Contamination from RL Post-training). We also run the tripartite perplexity comparison of Estimating Contamination via Perplexity, using pre- and post-cutoff FineWeb slices as its references.

The strongest existing competitor is ConStat (ConStat). It is performance-based, handles paraphrase leaks, and is the only method in the file reporting gaps on real models: $\delta \approx 8.3\%$ on GSM8K for Mistral-7b-v0.1. The file reports no per-item AUC for ConStat in a controlled-injection setting. To keep the comparison fair, ConStat will use the same paraphrase set that serves as our controls, and its benchmark-level gap will be compared with our mixture-estimated fraction. Self-Critique, which reports 0.70 AUC on Qwen2.5-7B-Instruct, is included as a secondary baseline on GSM8K, where answers are verifiable.

Two ablations isolate the key ingredients. The first removes the paraphrase control and uses the raw relative drop, which tests whether calibration is doing the work. The second replaces the mixture-fit cutoff with a fixed cutoff tuned on one model and transferred to others, which tests the DLT claim that self-calibrating thresholds matter. All methods get the same items, the same checkpoints, and the same small held-out benchmark for any tuning. Our probe's compute (forward and backward passes on a few thousand items) will be reported next to ConStat's inference cost.

Beating the baselines means two things. First, item-level AUC must exceed the best static detector by a margin larger than the seed-to-seed spread across three pre-training seeds, specifically at checkpoints where the contaminated-versus-clean accuracy gap has already vanished. Second, our estimated fraction must stay within a few points of the true 50% at those checkpoints, while ConStat's gap does not.

Why not just…?

Why not just score perplexity on the final checkpoint? Because the target phenomenon is defined by its invisibility to that measurement. Once the accuracy gap has "nearly vanished" after 25B clean tokens, there is little reason to expect a likelihood gap to survive. Static likelihood already collapses to chance under RL-stage contamination. The controlled study measures this directly: if static Min-K% matches our AUC at dormant checkpoints, the dynamic probe is unnecessary and we will report that.

Why not just use ConStat-style rephrasing tests? ConStat detects contamination as a failure to generalize from originals to rephrasings. In the dormant state, though, performance on the originals has itself returned to clean levels, so there should be no gap to measure. This is a real threat in partially dormant regimes, where some gap remains. The dormancy sweep, which varies the number of clean tokens after injection, will show where each method's detection power ends.

Why not just search the training corpus with embeddings? If the corpus is available, nearest-neighbor search against the benchmark is cheaper and should be preferred. We will run it in the controlled setting as an upper reference. N-gram filters are known to miss influential leaked tokens (Investigating Data Contamination for Pre-training Language Models), and embedding search avoids that weakness. The probe targets the common case where weights are released without the corpus. It also answers a different question: whether the model retained the item, which corpus presence alone does not establish.

Why not just run SFT and measure the inflation? Post-SFT gains mix real skill improvement with resurfacing. Without a clean twin model, which real audits never have, the two cannot be separated. The decisive check below tests whether the probe's per-item score predicts per-item SFT gain. If it does, the probe gives the same information without needing a counterfactual model.

Why not just use RADAR's mechanistic features? RADAR's features are read from a single forward pass and were validated on 100 test examples. Those features should be subject to the same dormancy problem as perplexity. We include RADAR's layer-convergence features as an additional static detector in the comparison.

Decisive Experiments

The study succeeds or fails on four checks. The central result is item-level AUC as a function of clean tokens after injection. Static detectors should decay toward chance as the leak goes dormant, while relearning score should stay high. If both decay together, the core claim is false. The dose-response check asks whether relearning score tracks repetition count non-monotonically, consistent with the curve the GPT-2 study found. The predictive check is the rank correlation between an item's relearning score and its accuracy gain under the SFT recipe of the post-training study, computed on held-out seeds. The confound check runs the probe on the twin model that was never contaminated: estimated contamination should be near zero there, and any residual measures how much item difficulty leaks past the paraphrase control. Only after these checks pass do we apply the probe to Mistral-7b-v0.1 on GSM8K and Llama-3-70b on ARC, alongside post-cutoff items as controls.