The foundational recipe for training large language models (LLMs) has long relied on a simple assumption: to scale a model efficiently, you should feed it a single, continuous stream of unique data. In their seminal work on compute-optimal scaling, Jordan Hoffmann et al. demonstrated in An empirical analysis of compute-optimal large language model training that parameters and tokens should scale in equal proportion. Crucially, this framework assumed a single-pass regime where the model encounters each token exactly once.
However, the AI field is rapidly approaching what researchers call the "data wall." As Pablo Villalobos et al. project in Position: Will we run out of data? Limits of LLM scaling based on human-generated data, the global stock of high-quality human-generated text data is finite and could be exhausted within the decade. To continue scaling, practitioners must reuse existing data across multiple training epochs.
This shift invalidates classical scaling laws. As Courtney Paquette et al. observe in 4+3 Phases of Compute-Optimal Neural Scaling Laws, multi-epoch training alters the compute-optimal frontier in ways single-pass models cannot predict. Even hyperparameter optimizations—such as the weight decay and batch size scaling laws mapped out by Shane Bergsma et al. in Power Lines: Scaling laws for weight decay and batch size in LLM pre-training—face significant uncertainty when data is recycled.
Understanding why and when data repetition works requires looking closely at how models accumulate information and why standard power laws eventually break down.
The Illusion of Freshness and Variance Accumulation
When a model is trained on a dataset for the first few epochs, repeated data behaves surprisingly well. In Improved Scaling Laws in Linear Regression via Data Reuse, Licong Lin et al. prove that during the early stages of multi-pass stochastic gradient descent (SGD), reusing data yields performance gains that closely match training on entirely fresh, unseen data points. The old data retains an "effective freshness" because the model's parameters have not yet fully extracted the underlying signal.
But this freshness is temporary. As training progresses into excessive passes, the predictable power-law scaling curves begin to collapse. The mechanistic reason for this breakdown lies in the model's error profile. In a single-pass regime, implicit regularization keeps the variance of the model's predictions in check. However, as Licong Lin et al. demonstrate in Scaling Laws in Linear Regression: Compute, Parameters, and Data, multi-pass training causes variance to accumulate rapidly.
Instead of learning generalizable structures, the model begins to overfit to the specific sample noise of the finite dataset. This variance accumulation causes the model's performance to deviate from standard scaling predictions, limiting the predictability of training trajectories (Shane Bergsma et al. in Scaling with Collapse: Efficient and Predictable Training of LLM Families) and fundamentally shifting how model capacity interacts with dataset size (Maissam Barkeshli et al. in On the Origin of Neural Scaling Laws: from Random Graphs to Natural Language).
Mathematical Penalties for Token Repetition
To account for this decay in utility, recent frameworks move away from simple token counts and instead model pretraining as a process of information accumulation.
In InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition, Weidong Zhou et al. formalize this by applying an exponential decay function to repeated data. Each time a model encounters a document it has seen before, the marginal information gain from that document drops exponentially. Fengze Liu et al. integrated this decay into LayerMix Law: Scaling Law for Large Language Models on Quality-Weighted Mixture Data with Repetition, allowing practitioners to accurately forecast performance drops in overtrained regimes rather than relying on expensive trial-and-error runs.
A similar mechanism is utilized by Shayne Longpre et al. in ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality. The authors introduce a data repetition saturation function that applies a smooth mathematical penalty to token counts once unique data is exhausted. This penalty reflects the hard boundary of data reuse: after a few epochs, the marginal return of repeating data flatlines, a constraint that even highly efficient architectures like Mixture-of-Experts (MoE) cannot escape without experiencing performance degradation on knowledge-intensive tasks (Houyi Li et al. in Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource).
The Size-Dependent Saturation Horizon
A common heuristic in machine learning has been to assume that the utility of repeating a dataset is capped at a fixed number of epochs (e.g., three to four passes), regardless of the dataset's size. Theoretical analysis has revealed that this static view is incorrect.
In Larger Datasets Can Be Repeated More: A Theoretical Analysis of Multi-Epoch Scaling in Linear Regression, Tingkai Yan et al. introduce the concept of the effective reuse rate. This metric quantifies the value of multi-epoch training by calculating the exact size of a unique dataset required to match the performance of a repeated dataset.
Linear Regime: [Pass 1] ──> [Pass 2] ──> [Pass 3] (Near-linear information gain)
│
▼
Decay Horizon: [Pass 4] ──> [Pass 5] ──> [Saturation] (Exponential decay of utility)
▲
└─ This horizon pushes outward as the unique dataset size grows.
The authors demonstrate that the saturation point—the moment when data repetition yields diminishing returns—scales dynamically as a function of the total number of unique samples. A small dataset contains fewer independent features and directions, meaning a model will quickly exhaust its novel combinations and begin memorizing noise.
Conversely, a larger dataset contains a vastly richer geometric structure. Because there are more unique data points to anchor the model's representations, the model can tolerate a significantly higher number of effective training epochs before the marginal utility of data reuse declines.
The Trade-Off Between Quality and Diversity
This size-dependent saturation fundamentally changes how we must approach data curation at scale. If high-quality data loses its utility exponentially with each repetition, when does it become better to switch to lower-quality data?
Pratyush Maini et al. address this in Scaling Laws for Data Filtering - Data Curation Cannot be Compute Agnostic. By modeling sample utility with an exponential half-life parameter, the authors show that aggressive data filtering is compute-dependent. While a pristine, highly filtered dataset provides an excellent signal initially, its value drops so sharply upon repetition that at large compute scales, seeing fresh, lower-quality data for the first time provides a stronger informational signal than seeing high-quality data for a third or fourth time.
Marginal Utility
▲
│ ┌─ High-Quality Repeated Data (Decays rapidly)
│ │
│ ▼
│ █░░░
│ █ ░░░
│ █ ░░░┌─ Fresh, Lower-Quality Data (Becomes superior at scale)
│ █ ░▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒
│ █ █ ▒ ▒
└──────────────────────────────► Total Compute / Epochs
This dynamic is supported by Ziheng Qin et al. in Dataset Growth, who demonstrate that as datasets reach high coverage, failing to prioritize diversity leads to rapid stagnation. As the scaling limits of both language and geometric models are pushed (Khang Ngo et al. in Scaling Laws and Symmetry, Evidence from Neural Force Fields), navigating the precise exchange rate between data freshness, repetition saturation, and dataset size becomes the core requirement for compute-optimal design.
Key Foundational Work
Scaling Data-Constrained Language Models (2023) — contributed 3 concept elements to this direction.
An empirical analysis of compute-optimal large language model training (2022) — contributed 1 concept elements to this direction.