Introduction
A core tenet of machine learning is that models should not be trained on their test sets. For Large Language Models (LLMs) trained on internet-scale data, however, benchmark datasets inevitably leak into the training corpus. The prevailing assumption is that this "data contamination" artificially inflates performance metrics and permanently invalidates evaluations.
But does a single encounter with a contaminated benchmark question early in training actually impact a final model's downstream performance? Modern LLMs are trained for hundreds of thousands of gradient steps over trillions of tokens. This paper challenges the assumption that any leakage permanently ruins evaluation validity. By analyzing scaling laws and optimizer dynamics, the authors demonstrate that in modern, data-rich training regimes, early contamination is often completely overwritten or "forgotten."
Methodology
To avoid the guesswork of post-hoc contamination detection, the authors trained models from scratch in a strictly controlled environment.
- Controlled Contamination: GPT-3 style models (up to 1.6B parameters) were trained on the 100B token split of the FineWeb-Edu dataset. The authors deliberately injected exact benchmark questions (from ARC-Easy, Social-I-QA, WinoGrande, PiQA, BoolQ, MMLU, and HellaSwag) into the training data at random positions.
- Scaling Three Dimensions: They measured benchmark overfitting by varying model size, the number of times a contaminated example is repeated (up to 144x), and the total number of training tokens.
- Continual Pre-training: To validate findings on larger models, they took intermediate checkpoints of OLMo-1B and OLMo-7B, injected contamination to cause an artificial accuracy spike, and then continued pre-training to measure the rate of forgetting.
- Optimizer Analysis: They mathematically and empirically analyzed the AdamW optimizer, specifically how the weight decay parameter forces the model to "forget" older gradient updates.
Key Findings: The Dynamics of Forgetting
The study reveals that the impact of data contamination is highly dependent on the training regime.
- The "Chinchilla" Vulnerability: If a model is trained strictly according to Chinchilla scaling laws (where training tokens are roughly 20x the parameter count), even minor contamination leads to significant overfitting. As model size increases under these constraints, the model's capacity to memorize leaked data grows.
- Data-Rich Regimes Wash Away Sins: Modern LLMs (like Llama 3) are trained on vastly more data than Chinchilla recommends (e.g., 10x to 15x more). In this regime, the effect of contamination vanishes. For instance, a 124M parameter model trained for 15x Chinchilla tokens completely forgot benchmark questions that were repeated 144 times early in training. Overfitting to contamination increases with model size and repetition, but decreases as the total number of training tokens grows.
- Rapid Recovery: Injecting benchmarks into an intermediate OLMo-7B checkpoint caused an immediate 17% artificial accuracy spike. However, continuing pre-training for just 13% of the remaining training time erased 87% of that artificial gain.
- Forgetting Requires Novel Data: Models do not forget if they are simply trained on the same data for multiple epochs. True forgetting requires the continuous stream of novel data characteristic of LLM pre-training.
- Weight Decay Drives Forgetting: The weight decay parameter in the AdamW optimizer actively bounds and accelerates empirical forgetting. By analyzing the cumulative weight decay of large-scale training runs, the authors estimate that models like Llama 3 405B have likely completely forgotten the first 10% of their training data.
Limitations and Future Directions
While these findings provide a new perspective on data contamination, the authors note several limitations. The study focused on exact (verbatim) contamination; it does not address whether models forget paraphrased or translated benchmark questions, which larger models might internalize as general concepts. Furthermore, the forgetting dynamics observed apply to benchmark questions and should not be extrapolated to random strings or Personally Identifiable Information (PII) in privacy contexts. Finally, while OLMo-7B was used for continual pre-training, the from-scratch controlled experiments maxed out at 1.6B parameters, leaving the exact scaling curves for >70B parameter models for future work.
Summary updated