When a language model is trained on the test questions of a benchmark, its score goes up, and a family of detectors tries to catch this after the fact. Contamination studies, and the papers that rank detectors against each other, almost always train one contaminated model per condition and report one number. This project asks whether that number is reproducible. If we retrain the same model on the same leaked data and change only the random seed (the random number that sets the initial adapter weights, the order of training examples, and the sampling during reinforcement learning), do we get the same score inflation and the same detector accuracy? There is good reason to doubt it. Fine-tuning a medical model under one seed produced heavy forgetting of general knowledge, and under two other seeds it produced none (Capacity-Gated Forgetting in LoRA Fine-Tuning). It would be surprising if what a model memorizes were much more stable than what it forgets.
We propose a controlled study plus a cheap measurement protocol. The protocol trains contaminated and clean "twin" models: both share every random seed, and they differ only in whether the benchmark items were injected into the training data. Subtracting a clean twin from its contaminated sibling cancels most of the seed-driven noise. That lets us state, with honest error bars, how much contamination inflates scores, how much reinforcement learning hides it, and whether published gaps between detectors are larger than the seed lottery. In two sentences: contamination effects and detector accuracies may vary across training seeds by amounts comparable to the differences the field reports. Pairing each contaminated run with a same-seed clean twin is a simple way to measure and remove that variance.
Core Claim
Our hypothesis has three parts. First, across training seeds, the spread in score inflation from fine-tuning contamination, and in the accuracy of detectors that try to flag it, is of the same order as the effects reported in single-seed studies. Those effects include the 8.8% inflation in pass@1 (the fraction of problems solved on the first attempt) and the roughly 14 to 16 point drop in detector AUROC after reinforcement learning reported in On The Fragility of Benchmark Contamination Detection in Reasoning Models. AUROC is the area under the ROC curve, where 0.5 is chance and 1.0 is perfect separation. Second, which individual items get memorized depends partly on the seed and is not only a property of the item. Third, same-seed twin pairing removes most of this variance at the cost of one extra run per seed. The result is tighter confidence intervals than averaging the same number of unpaired runs.
The twin estimator works as follows. Let $A_c(s)$ and $A_0(s)$ be benchmark accuracy for the contaminated and clean model trained under seed $s$. Let $\sigma^2$ be the seed variance of either one, and let $r$ be the correlation between twins that share a seed. Then
$$\mathrm{Var}\big[A_c(s)-A_0(s)\big] = 2\sigma^2(1-r),$$
whereas comparing independently seeded runs gives $2\sigma^2$. The pairing is worth exactly as much as $r$ is large. The same construction applies to any per-item detector score, and we measure $r$ directly. If $r$ turns out near zero, pairing buys nothing, and that is itself a finding about how chaotic fine-tuning is.
Relation to Prior Work
Four lines of work motivate the proposal. The seed-sensitivity literature shows that the seed alone flips model rankings (Reporting Score Distributions Makes a Difference), can move retrieval recall by up to 10 points (Where Does the Performance Improvement Come From?), and reshapes deep-layer representations even when accuracy is unchanged (Individual differences among deep neural network models). None of these papers studies memorization of evaluation data.
The forgetting study by Narula et al. is the closest seed-sensitivity result in large language model fine-tuning. Its seeds 0 and 7 also changed batch size and sequence length, so the seed effect is confounded. Our design varies one random source at a time and holds everything else fixed.
On the contamination side, When Is Benchmark Contamination Detectable? bounds audit power by the product of the contamination fraction, behavioral separability, and the square root of the audit size. It does train five seeds, but it uses them to build causal controls, not to ask whether separability itself moves with the seed. We treat separability as a random quantity indexed by the seed. The detector papers (Fragility, Detecting Data Contamination from Reinforcement Learning Post-training, LaRA) report single-run AUROCs without seed intervals. These are the results whose stability we test.
LaRA deserves special attention because it reads contamination off layer-wise representation geometry. Mehrer et al. show that this geometry is exactly what varies across seeds. If LaRA's signal survives reseeding, that is strong evidence it is real. If it does not, it is the detector most at risk.
Can Training Logs Make Model Comparisons More Precise? supplies a complementary variance-reduction technique: regression on validation loss measured early in training. It also supplies a constraint. That adjustment needs roughly 15 to 20 runs before it helps. Twin pairing needs no fitted coefficients, so it should still work at the small run counts that large language model budgets allow.
Research Program
We train small and mid-size open models, in the 0.5B to 1.5B range from families such as Qwen2.5 and SmolLM2 studied in Sensitivity of Small Language Models to Fine-tuning Data Contamination, with a 7B confirmation run. Each model goes through the two-stage pipeline of the Fragility paper. Stage one is supervised fine-tuning, with or without injected benchmark items. Stage two is clean GRPO, a reinforcement learning method that scores groups of sampled answers against each other.
The contaminated items come from the RL-MIA math split (AIME problems, half of which are injected) and from MedQA-USMLE questions. We separate three random sources and vary each independently: adapter initialization, data order, and rollout sampling during reinforcement learning.
For each condition we run 16 twin pairs at small scale and 4 at 7B. We record the full item-by-seed matrix of correctness and of detector scores. This lets us decompose variance into item, seed, and item-by-seed components. We report intraclass correlation (the share of variance explained by which item it is, as opposed to which seed) as the measure of how item-intrinsic memorization is. The detectors are loss, Min-K% probability (which averages the least likely tokens in a passage), LiRA (a membership test calibrated with reference models), Self-Critique entropy, the LaRA geometric scores, and the CAP consistency ratio (CAP).
Baselines
The naive reference point is the single-seed protocol that current papers use: one contaminated model and one clean model trained under different or unrecorded seeds. We report its result as a distribution over our 16 seeds rather than as one number. Concretely, we report how often a single-seed study would have reached the wrong conclusion about whether a detector beats chance or whether reinforcement learning cloaks contamination.
The strongest existing estimator of the same quantities is unpaired multi-seed averaging. It uses exactly the same number of total runs, the same data, and the same hyperparameters as the twin design, and differs only in that the clean and contaminated seeds are drawn independently. The twin protocol beats it if its 95% intervals on score inflation and on change in detector AUROC are narrower at equal compute. The effect we are hoping for is narrower by at least a third, which corresponds to $r \geq 0.55$ in the equation above.
We also compare against early-training-log covariate adjustment, following Huang's cross-fitted recipe on validation loss at one third of training. We run it alone and combined with pairing, at the same run counts.
For detector accuracy itself, the strongest reported numbers are as follows. LiRA reaches 89.1% AUROC after supervised contamination, falling roughly 14 points after one epoch of GRPO (Fragility). Self-Critique reaches 0.70 AUC on RL-MIA with Qwen2.5-7B-Instruct (Detecting Data Contamination from Reinforcement Learning Post-training). LaRA combined with Self-Critique reaches 0.79 ROC-AUC on a 60-sample set (LaRA). We are not trying to beat these numbers. We ask whether they fall inside or outside our seed intervals.
The ablation that isolates the key ingredient is a twin design in which the clean twin shares only the initialization seed, or only the data-order seed. This identifies which random source the pairing is actually cancelling.
Why not just…?
Why not just average three to five seeds? This is the most serious alternative. If seed variance is roughly Gaussian and moderate, it may be all anyone needs. The forgetting results suggest the outcome can instead be bimodal: a mean of 0.352, −0.015 and −0.015 describes none of the three runs. Our unpaired-averaging baseline tests this directly at equal compute. If it matches twin pairing, we will recommend it and say so.
Why not just rely on LiRA, whose reference models already absorb training randomness? LiRA uses models trained under many seeds to calibrate its decision for each item. That handles randomness in the reference models, not variance in the audited model or in the reported AUROC. LiRA also loses most of its signal after GRPO. We measure LiRA's AUROC spread across seeds of the audited model, which its own design cannot reveal.
Why not just use contamination-limited benchmarks like LiveBench? LiveBench avoids leakage for tasks whose answers a program can check, refreshing about a sixth of its questions each month. It cannot audit existing models on MMLU or medical boards. It also says nothing about how reliable the detectors are, so it complements this work rather than replacing it.
Why not just use perturbation consistency tests, which need no retraining? CAP-style checks are cheap. But the Fragility paper reports that contamination with step-by-step solutions generalizes across paraphrases, leaving detectors near chance. CAP's ratio is also a single-seed statistic. We include it as a detector and measure its seed spread like the others.
Decisive Checks
The hypothesis fails if three things all hold across seeds: the seed standard deviation of detector AUROC stays under 2 points, detector rankings never flip, and GRPO's AUROC reduction appears in every seed with similar size. In that case single-seed reporting is adequate, and we would publish that reassurance.
It succeeds if any of the following holds: the seed intervals overlap the reported gaps between detectors, some seeds keep a detectable loss gap after GRPO while others do not, or LaRA's geometric signal loses its rank ordering across reinforcement learning rollout seeds. The pairing contribution is supported only if the measured twin correlation $r$ makes paired intervals substantially tighter than unpaired averaging at equal compute.
References
- Capacity-Gated Forgetting in LoRA Fine-Tuning: Rank, Proximity, and Endogenous Replay in Medical LLMs
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging
- Where Does the Performance Improvement Come From? - A Reproducibility Concern about Image-Text Retrieval
- Individual differences among deep neural network models
- Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
- LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Training
- Can Training Logs Make Model Comparisons More Precise?
- Sensitivity of Small Language Models to Fine-tuning Data Contamination
- CAP: Data Contamination Detection via Consistency Amplification
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark