How Much Can We Forget about Data Contamination?

Sebastian Bordt, Suraj Srinivas, Valentyn Boreiko, Ulrike von Luxburg

Introduction

A core tenet of machine learning is that models should not be trained on their test sets. For Large Language Models (LLMs) trained on internet-scale data, however, benchmark datasets inevitably leak into the training corpus. The prevailing assumption is that this "data contamination" artificially inflates performance metrics and permanently invalidates evaluations.

But does a single encounter with a contaminated benchmark question early in training actually impact a final model's downstream performance? Modern LLMs are trained for hundreds of thousands of gradient steps over trillions of tokens. This paper challenges the assumption that any leakage permanently ruins evaluation validity. By analyzing scaling laws and optimizer dynamics, the authors demonstrate that in modern, data-rich training regimes, early contamination is often completely overwritten or "forgotten."

Methodology

To avoid the guesswork of post-hoc contamination detection, the authors trained models from scratch in a strictly controlled environment.

  • Controlled Contamination: GPT-3 style models (up to 1.6B parameters) were trained on the 100B token split of the FineWeb-Edu dataset. The authors deliberately injected exact benchmark questions (from ARC-Easy, Social-I-QA, WinoGrande, PiQA, BoolQ, MMLU, and HellaSwag) into the training data at random positions.
  • Scaling Three Dimensions: They measured benchmark overfitting by varying model size, the number of times a contaminated example is repeated (up to 144x), and the total number of training tokens.
  • Continual Pre-training: To validate findings on larger models, they took intermediate checkpoints of OLMo-1B and OLMo-7B, injected contamination to cause an artificial accuracy spike, and then continued pre-training to measure the rate of forgetting.
  • Optimizer Analysis: They mathematically and empirically analyzed the AdamW optimizer, specifically how the weight decay parameter forces the model to "forget" older gradient updates.

Key Findings: The Dynamics of Forgetting

The study reveals that the impact of data contamination is highly dependent on the training regime.

Scaling effects on data contamination
  • The "Chinchilla" Vulnerability: If a model is trained strictly according to Chinchilla scaling laws (where training tokens are roughly 20x the parameter count), even minor contamination leads to significant overfitting. As model size increases under these constraints, the model's capacity to memorize leaked data grows.
  • Data-Rich Regimes Wash Away Sins: Modern LLMs (like Llama 3) are trained on vastly more data than Chinchilla recommends (e.g., 10x to 15x more). In this regime, the effect of contamination vanishes. For instance, a 124M parameter model trained for 15x Chinchilla tokens completely forgot benchmark questions that were repeated 144 times early in training. Overfitting to contamination increases with model size and repetition, but decreases as the total number of training tokens grows.
  • Rapid Recovery: Injecting benchmarks into an intermediate OLMo-7B checkpoint caused an immediate 17% artificial accuracy spike. However, continuing pre-training for just 13% of the remaining training time erased 87% of that artificial gain.
Cross-entropy loss difference showing rapid forgetting
  • Forgetting Requires Novel Data: Models do not forget if they are simply trained on the same data for multiple epochs. True forgetting requires the continuous stream of novel data characteristic of LLM pre-training.
  • Weight Decay Drives Forgetting: The weight decay parameter in the AdamW optimizer actively bounds and accelerates empirical forgetting. By analyzing the cumulative weight decay of large-scale training runs, the authors estimate that models like Llama 3 405B have likely completely forgotten the first 10% of their training data.

Limitations and Future Directions

While these findings provide a new perspective on data contamination, the authors note several limitations. The study focused on exact (verbatim) contamination; it does not address whether models forget paraphrased or translated benchmark questions, which larger models might internalize as general concepts. Furthermore, the forgetting dynamics observed apply to benchmark questions and should not be extrapolated to random strings or Personally Identifiable Information (PII) in privacy contexts. Finally, while OLMo-7B was used for continual pre-training, the from-scratch controlled experiments maxed out at 1.6B parameters, leaving the exact scaling curves for >70B parameter models for future work.

Summary updated

Abstract

The leakage of benchmark data into the training data has emerged as a significant challenge for evaluating the capabilities of large language models (LLMs). In this work, we challenge the common assumption that small-scale contamination renders benchmark evaluations invalid. First, we experimentally quantify the magnitude of benchmark overfitting based on scaling along three dimensions: The number of model parameters (up to 1.6B), the number of times an example is seen (up to 144), and the number of training tokens (up to 40B). If model and data follow the Chinchilla scaling laws, minor contamination indeed leads to overfitting. At the same time, even 144 times of contamination can be forgotten if the training data is scaled beyond five times Chinchilla, a regime characteristic of many modern LLMs. Continual pre-training of OLMo-7B corroborates these results. Next, we study the impact of the weight decay parameter on example forgetting, showing that empirical forgetting occurs faster than the cumulative weight decay. This allows us to gauge the degree of example forgetting in large-scale training runs, indicating that many LLMs, including Llama 3 405B, have forgotten the data seen at the beginning of training.
BibTeX
@inproceedings{bordt2025how,
  author = {Sebastian Bordt and Suraj Srinivas and Valentyn Boreiko and Ulrike von Luxburg},
  title = {{How Much Can We Forget about Data Contamination?}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=Pf0PaYS9KG},
}
Figure