Scale Dependent Data Duplication

Joshua Kazdan, Noam Itzhak Levi, Rylan Schaeffer, Jessica Chudnovsky, Abhay Puri, Bo He, Mehmet Donmez, Sanmi Koyejo, David L. Donoho

TL;DR

This paper reveals that data duplication isn't just a property of a dataset—it’s a property of the model training on it. As language models grow more capable, they begin to treat semantically similar documents (like translations or paraphrases) as redundant "duplicates," causing traditional scaling laws to break down and highlighting a looming "diversity crisis" in web-scale and synthetic data.

What Problem Does This Paper Address?

In the race to train ever-larger Large Language Models (LLMs), the industry has relied on two assumptions: first, that we can "deduplicate" data by removing near-identical text matches; and second, that model performance scales predictably with more data.

However, the authors argue that at web scale, the definition of a "duplicate" is fuzzy. If a model is smart enough to understand that an English sentence and its French translation convey the same information, does training on both provide a new signal, or is it just redundant noise? As we approach the limits of unique human-generated data on the internet, understanding how "semantic redundancy" affects scaling is critical for predicting the performance of the next generation of frontier models.

Methods

The researchers tackled this problem through three distinct experimental lenses:

  1. Gradient Alignment: They measured the cosine similarity between the gradients (the "learning signals") produced by a document and its transformed versions (e.g., translated, re-cased, or paraphrased). If the gradients are highly aligned, the model is effectively treating the two versions as the same example.
  2. Embedding Analysis: They embedded 192 million documents from the FineWeb-Edu dataset using a state-of-the-art embedding model (EmbeddingGemma-300m) to analyze "semantic collisions"—instances where different documents are extremely close in meaning.
  3. Controlled Scaling Ladders: They trained a suite of models (34M to 344M parameters) on data streams sampled from finite pools of unique documents. This allowed them to isolate how repetition affects models of different sizes.
  4. Theoretical Modeling: They derived a new "Plane Law" for scaling that incorporates "effective uniqueness" ($K_{eff}$), allowing practitioners to predict performance drops caused by limited data diversity.

Datasets

  • FineWeb-Edu-Dedup: A high-quality subset of web data containing 192 million documents.
  • Recycling-the-Web: A 44-million-document fully-synthetic pretraining corpus used to compare the diversity of AI-generated data against real-world data.

Key Insights and Ideas

  • Duplicates are Scale-Dependent: Small models are "surface-level" learners; they see a translation as entirely new data. Large, capable models are "semantic" learners; they recognize the underlying meaning, which makes semantically similar data act like a duplicate.
  • The Semantic Collision Wall: In small datasets, the distance between "nearest neighbor" documents follows a predictable power law. However, once a dataset exceeds roughly 10 million documents, the number of semantic collisions (near-duplicates in meaning) explodes much faster than expected.
  • The Synthetic Data Warning: Synthetic datasets (AI-generated) hit the "diversity wall" an order of magnitude earlier than human data. This suggests that simply generating more tokens with LLMs won't solve the data exhaustion problem unless we find ways to "seed" more semantic diversity.
  • Scaling Laws Break Down: Naive scaling laws over-predict performance for large models when data uniqueness is limited. A large model trained on a repetitive dataset suffers a much harsher "loss penalty" than a small model on that same dataset.

Key Results

  • Gradient Sensitivity: As model parameters increase from $10^7$ to $10^{10}$, the ability to distinguish a document from its translation (measured by AUC) rises significantly, proving that smarter models "see" the redundancy.
  • Collision Acceleration: Nearest-neighbor similarities deviate sharply from isotropic baselines at large scales, indicating that the "semantic space" of human thought is more crowded than we assumed.
  • Restored Predictivity: The authors' proposed 3-parameter scaling law (Equation 28) accurately predicted evaluation losses across different compute budgets and data pool sizes, achieving a mean absolute relative error of only ~0.90%.

Limitations and Future Work

The authors acknowledge that due to compute constraints, they simulated semantic collisions using exact document repeats in their scaling experiments. While they provide strong evidence that semantic similarity induces the same effects as exact repeats, future work needs to validate this on multi-billion parameter models trained on truly semantically-deduplicated vs. raw datasets.

Additionally, the study relies on a specific embedding model (EmbeddingGemma-300m). While a top-tier model, the authors note that better embeddings that more efficiently utilize latent space could refine these results. The ultimate goal is to develop "semantic deduplication" pipelines that can identify and prune redundancy at the level of ideas, not just characters.

Summary updated

Abstract

Data duplication during pretraining can degrade generalization and lead to memorization, motivating aggressive deduplication pipelines. However, at web scale, it is unclear what constitutes a "duplicate": beyond surface-form matches, semantically equivalent documents (e.g. translations) may induce redundant training signals once models become sufficiently capable. Practically, this means that semantic duplicates operate increasingly like exact duplicates during training. We present evidence that duplication is scale-dependent in two ways. First, as model capability increases, cross-entropy loss gradients for semantically equivalent documents become more aligned. Smaller models, by contrast, produce gradients that reflect surface similarity (e.g., shared tokens) rather than semantic similarity. Second, we embedded all 192 million FineWeb-Edu-Dedup documents using EmbeddingGemma-300m. For moderate corpus sizes, the cosine similarity between nearest-neighbors follows an isotropic power law baseline. However, as corpus size grows to hundreds of billions of tokens, the nearest-neighbor similarities deviate sharply, indicating accelerated semantic collisions. Finally, controlled pretraining on data sampled with replacement from pools of finite unique documents shows that limited uniqueness yields mild degradation for small models, but rapidly increasing loss penalties for larger models, breaking naive scaling extrapolation. We derive explicit scaling laws that allow practitioners to estimate deviation from expected scaling due to limited semantic uniqueness of the pretraining corpus. Our results identify and resolve an unstudied source of scale-dependence, allowing for more accurate prediction at scale.
BibTeX
@inproceedings{kazdan2026scale,
  author = {Joshua Kazdan and Noam Itzhak Levi and Rylan Schaeffer and Jessica Chudnovsky and Abhay Puri and Bo He and Mehmet Donmez and Sanmi Koyejo and David L. Donoho},
  title = {{Scale Dependent Data Duplication}},
  booktitle = {MemFM @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=N835H9OajI},
}
Figure