TL;DR
Data contamination is widely feared to ruin large language model (LLM) evaluations, but this paper reveals a significant silver lining: modern, overtrained models naturally "forget" contaminated data over time. By scaling training data well beyond the Chinchilla-optimal limit, models can completely erase the performance inflation caused by seeing benchmark questions up to 144 times. This natural amnesia is heavily driven by the mathematical dynamics of weight decay in standard optimizers like AdamW.
What Problem Does This Paper Address?
Data contamination—where evaluation benchmarks accidentally leak into the massive internet-scale pre-training datasets of LLMs—is a major challenge for AI evaluation. The standard assumption within the community is that any amount of contamination permanently invalidates benchmark results due to overfitting.
However, modern foundation models are frequently "overtrained," running for millions of gradient steps on trillions of tokens that far exceed traditional Chinchilla scaling laws. The authors tackle a fundamental question: Does a single update on contaminated data early in training permanently alter the final model's downstream performance, or do LLMs naturally forget these leaked examples as training progresses?
Methods
The authors study data contamination in a highly controlled setting by training GPT-3 architecture models from scratch (up to 1.6B parameters) and explicitly injecting exact benchmark questions at random positions within the training stream. They systematically vary three dimensions: model size, the number of training tokens, and the number of times an example is repeated.
To validate their findings on larger scales, they perform continual pre-training experiments using intermediate checkpoints of the open-source OLMo-1B and OLMo-7B models. Finally, they mathematically analyze the optimization dynamics of the AdamW optimizer, tracking how cumulative weight decay acts as an upper bound for how much a model retains past gradient updates.
Datasets
- Pre-training Data: The 100-billion-token (100BT) split of the FineWeb-Edu dataset.
- Evaluation/Contamination Benchmarks: A balanced mix of seven popular language modeling benchmarks: ARC-Easy, Social-I-QA, WinoGrande, PiQA, BoolQ, MMLU, and HellaSwag.
- Data Cleaning: The authors implemented rigorous filtering using fuzzy string matching (Levenshtein distance) to remove pre-existing near-duplicates within and across these benchmarks, ensuring their controlled contamination measurements were accurate.
Key Insights and Ideas
- Overtraining Cures Contamination: While Chinchilla-optimal models suffer heavily from minor contamination, scaling the training data to 5x or more of the Chinchilla limit causes the model to completely forget the leaked data.
- Spaced Repetition Helps LLMs Remember: Interestingly, benchmark questions distributed uniformly throughout training cause stronger final overfitting than questions clustered exclusively at the very end of training. Just like humans, spaced repetition helps LLMs retain information.
- Weight Decay is a Forgetting Engine: Forgetting isn't just a passive side effect of seeing new data; it is causally driven by the weight decay parameter ($\lambda$) and learning rate schedule ($\gamma$) in AdamW. Empirically, models actually forget past examples faster than the theoretical bound calculated from cumulative weight decay.
- Implications for Frontier Models: By analyzing the optimization hyperparameters of massive models like Llama 3 405B, the authors show that data seen in the first 10% of training is mathematically guaranteed to be forgotten by the end of the run.
Key Results
- For a 124M parameter model trained at 2x Chinchilla, repeating a contaminated dataset 12 times inflated benchmark accuracy by 18 percentage points. However, when trained at 15x Chinchilla, that accuracy difference dropped to zero—complete empirical forgetting.
- In the OLMo-7B experiments, injecting benchmark questions 4 times initially boosted accuracy by an average of 17 percentage points. Continuing pre-training for just 13% more of the original schedule erased 87% of this artificial performance gain.
Limitations and Future Work
- Exact vs. Non-Exact Contamination: The study focuses strictly on verbatim text matching. It does not evaluate paraphrased questions or translated text, which larger models might generalize and retain differently.
- Scale Constraints: While the authors successfully extended their findings to a 7B parameter model via continual pre-training, training 100B+ parameter models from scratch under controlled contamination remains computationally prohibitive.
- Benchmark Data vs. Privacy Data: The results are restricted to natural language benchmark questions. The authors caution that empirical forgetting might behave differently for random strings, code, or uniquely identifiable personal information, meaning these insights shouldn't be blindly extrapolated to privacy or machine unlearning contexts.
Summary updated