Auditing Benchmark Contamination as a Panel Study Over Training Checkpoints

When a language model is trained on the test questions it will later be graded on, its benchmark score stops measuring ability. This is called benchmark contamination. Current tools for catching it look at a single finished model, and that is increasingly a losing position. Post-training with reinforcement learning erases the statistical traces these detectors rely on: after one epoch of clean reinforcement learning, a detector that had caught fine-tuning contamination with 89.1% AUROC (area under the ROC curve, where 50% is chance) falls to near chance, according to On The Fragility of Benchmark Contamination Detection in Reasoning Models. Paraphrasing the leaked questions also removes the before-and-after-cutoff performance drop that temporal benchmarks depend on, as Test of Time showed. The finished model is a single snapshot, and a snapshot can be cleaned up.

We propose to treat a training run the way social scientists treat a panel survey, which follows the same people over many interview waves. Here the "respondents" are benchmark items and the "waves" are saved training checkpoints. Each benchmark item is paired with a matched paraphrased twin that was never trained on. We then follow the gap between item and twin across checkpoints. Shared capability growth moves both members of a pair together. Contamination shows up as an item-specific jump in that gap at some checkpoint. Later reinforcement learning may shrink the gap in the final model, but it cannot remove the jump from the recorded history. The two-sentence version: contamination is an event in the training history, not a property of the final weights, so audit the history with each item compared against its own clean twin.

Core Claim

Given a sequence of checkpoints, a paired item-versus-twin trajectory test detects contamination where final-model detectors fail. These failures include contamination that reinforcement learning has later hidden, contamination injected into chain-of-thought fine-tuning of an existing reasoning model, and contamination at low fractions. The test also says when in training the leak entered.

The method comes from panel econometrics. Online and Social Media Data As an Imperfect Continuous Panel Survey treats social media streams as imperfect panels so that population-level change can be split into new members arriving and existing members intensifying. Our analogue splits a benchmark's score gain into two parts. One part is shared with matched twins: general capability, including items that become solvable for legitimate reasons. The other part is item-specific excess gain, which is the contamination signal. The privacy literature supplies a reason to expect this signal to persist. Work on recommendation access patterns (Private Data Leakage via Exploiting Access Patterns of Sparse Features) and browsing histories (Protecting User Privacy) shows that individual trajectories stay identifiable after surface identifiers are scrubbed. We hypothesize the same holds for items: the shape of an item's history survives even when its final-state signal has been compressed.

Concretely, let $y_{it}$ be a per-item score at checkpoint $t$, such as the mean log-probability of the reference answer or the fraction of $k$ samples that are correct. Let $g_{it} = y_{it} - y_{i't}$ be the gap between item $i$ and its twin $i'$. We fit

$$g_{it} = a_i + b_t + \varepsilon_{it},$$

where $a_i$ is a fixed per-pair offset that absorbs any built-in difficulty difference from paraphrasing, $b_t$ is a per-checkpoint effect shared by all pairs, and $\varepsilon_{it}$ is residual noise. The per-item test statistic is the largest standardized change point in the residuals $\varepsilon_{it}$. We calibrate it against a null distribution built from pairs known to be clean, such as items released after the base model and never injected. The argmax checkpoint gives the estimated leak time.

Relation to Prior Work

Three papers motivate the gap. To the Cutoff... and Beyond? uses model cutoff dates as a natural experiment. It finds that GPT-4's Codeforces pass rate falls from 37% to 16% across the cutoff, but it evaluates one frozen model on pass/fail outcomes. Test of Time shows that this kind of temporal signal can be laundered by rephrasing. It verifies the point with influence functions, which trace answers back to training documents, but their cost limited the audit to 40 papers. On The Fragility of Benchmark Contamination Detection in Reasoning Models explains why reinforcement learning hides contamination: clipping in the policy-update step compresses the loss gap between seen and unseen items. The authors call for releasing intermediate checkpoints but do not provide a way to audit them. Our proposal is that missing method.

Two papers supply technique. PaCoST runs a paired significance test of model confidence on original versus rephrased items, but only on one model. We reuse its pairing and extend it along the training-time axis. When Is Benchmark Contamination Detectable? bounds detection power by $\alpha\rho\sqrt{m}$, where $\alpha$ is the contaminated fraction, $\rho$ is how separable contaminated and clean behavior are, and $m$ is the number of items. It also shows that naive sample-size rules fail at small $m$. We use its two-stage simulation planner to set audit sizes. Our working hypothesis is that checkpoint history raises the effective $\rho$, because a jump is more separable than a compressed endpoint. This will be measured, not assumed.

Two further papers are near neighbors that our design must differ from. CANARY takes the difference in mid-layer hidden states between a base model and a tuned model and filters it through a sparse autoencoder (a model that breaks activations into a small number of interpretable features). It catches 1% fine-tuning poisoning at AUROC 1.000, but it compares only two checkpoints, needs a pristine base, and was tested on medical-advice poisoning rather than benchmarks. ConStat flags non-generalizing benchmark performance by comparing against reference models on twin benchmarks. That gives benchmark-level verdicts at a single time point, whereas ours are item-level and longitudinal.

Research Program

We first build contaminated training histories with known ground truth, following the protocols of the Fragility paper and the information-limits paper.

Stage I. Qwen2.5-7B-Instruct is fine-tuned on a mix that includes a known subset of AIME, AMC, GPQA Diamond, OlympiadBench and Minerva Math items. It is then trained with clean GRPO, a reinforcement learning method that scores each answer relative to a group of sampled answers. We save dense checkpoints throughout both phases.

Stage II. DeepSeek-R1-Distill-Llama-8B is fine-tuned on contaminated chain-of-thought data. This is the setting where all ten detectors in the Fragility paper scored 50–58% AUROC.

Pre-training regime. We follow the controlled injection of SQuAD into the Pile with Pythia, GPT-Neo and similar models.

Variable conditions. Across these settings we vary three things: the contaminated fraction $\alpha$; the exposure type, following the separability ordering of verbatim, then paraphrase, then surface perturbation, then answer-only; and when contamination enters, including a hard case where leaked items are mixed into the reinforcement learning prompts themselves.

Twins. For each item we generate a paraphrased twin. We hold out a set of never-injected pairs, which provides the clean null. We also use GSM8k with its human-written mirror GSM1k (A Careful Examination of LLM Performance on Grade School Arithmetic) as a twin pair whose construction does not depend on an LLM paraphraser.

Per-item signals. We record three signals at each checkpoint, so we can see which one carries the history signal:

  • answer log-probability, which needs white-box access;
  • P(True) confidence in the style of PaCoST;
  • sampled correctness, which needs only generated text.

Baselines

Naive reference. The do-nothing point is final-checkpoint benchmark accuracy compared against twin accuracy, a single-snapshot score gap. The Fragility paper reports this gap as the 8.8% Stage I and 11.7% Stage II inflation. It shows that inflation exists but gives no item-level verdict.

Strongest existing detector. LiRA, a likelihood-ratio membership test that uses reference models, reports 89.1% AUROC on Stage I before reinforcement learning, roughly 14 points lower after one GRPO epoch, and 50–58% AUROC for all ten detectors in Stage II (On The Fragility…). We run LiRA, Min-K%, and loss in two ways: on the final checkpoint, and on every checkpoint, keeping the best checkpoint chosen in hindsight. The every-checkpoint variant is an oracle-favored version and the comparison that matters most.

Other detectors. PaCoST is run per checkpoint on the same twins. For PaCoST and ConStat on these specific checkpoint sequences, no reported result was found on Lacuna, so we run them ourselves. CANARY is run between consecutive checkpoint pairs. It reports AUROC 1.000 at 1% contamination in its own medical-poisoning setting, but no reported result was found on Lacuna for it on benchmark contamination.

Fairness. All methods receive the same checkpoints, the same items and twins, the same forward-pass budget, and thresholds tuned on the same held-out clean pairs.

Ablations. Three ablations isolate the key ingredients:

  • removing the twin, so the item trajectory is analyzed alone;
  • removing history, so only two checkpoints are compared;
  • removing the per-checkpoint effect $b_t$, so the shared-capability correction is gone.

What counts as winning. Beating the baselines means two things. First, item-level AUROC at least 10 points above best-checkpoint LiRA in Stage II and in the contamination-during-RL condition. Second, no worse than LiRA where LiRA already works, which is Stage I before reinforcement learning.

Why not just…?

Why not just run LiRA on the last checkpoint before reinforcement learning? This is the real threat. In clean Stage I, where contamination enters only during fine-tuning and a pre-RL checkpoint is available, LiRA's 89.1% may be as good as anything we build. In that case the trajectory method adds only leak timing. We report this honestly. The proposal's claim rests on the settings where no clean "before-cloaking" checkpoint exists. In Stage II, detectors fail on the contaminated model itself. In the RL-mixing condition, leakage and cloaking happen together. The best-checkpoint baseline directly tests whether history adds anything beyond choosing the right snapshot.

Why not just compare benchmark accuracy against a twin benchmark, as ConStat or GSM1k do? Those methods give a benchmark-level inflation estimate, but they cannot say which items leaked or when. They also mix contamination together with distribution mismatch between the original benchmark and its twin. The per-pair offset $a_i$ in our model absorbs fixed mismatch. The aggregate-gap baseline above measures how much item-level resolution buys.

Why not just use temporal cutoff splits? Test of Time shows that rephrased questions show no post-cutoff decay; physics scores were even +2.19 points higher after the cutoff. Cutoffs also do not apply to controlled training runs, where the leak is injected at a known step. We include cutoff-split accuracy only on the GSM8k/GSM1k and Codeforces-style analyses, as a sanity reference.

Decisive Checks

The central experiment asks whether, in Stage II, the paired trajectory test separates injected items from clean ones well above the 50–58% range, and whether the estimated leak checkpoint matches the true injection step. A negative result here would sink the core claim.

The second check is power. Using the planner from the information-limits paper, we measure power against $\alpha$, $m$, and the number of checkpoints $T$. We ask whether more checkpoints buy power or whether the checkpoint noise is too correlated to help. This tells developers how many checkpoints are worth releasing.

The third check is the exposure hierarchy. If the history signal falls to chance for answer-only exposure, as single-snapshot detectors do, we will report that as the method's boundary.

Finally, we test a text-only variant using sampled correctness alone. This checks whether the approach transfers to sequences of public model versions accessed through an API, where log-probabilities are unavailable.