Scaling Language Models Beyond the Human Data Wall

49 concepts43 papers1 proposal

The compute-optimal training recipe established by Jordan Hoffmann et al. in An empirical analysis of compute-optimal large language model training dictates a strict linear relationship: to scale a model efficiently, the volume of training tokens must grow in proportion to the number of model parameters. For years, this meant scraping increasingly massive swaths of the public internet.

However, we are rapidly approaching a physical limit. Pablo Villalobos et al. demonstrate in Position: Will we run out of data? Limits of LLM scaling based on human-generated data that the global supply of high-quality, human-generated public text is finite and projected to be exhausted between 2026 and 2032. This impending shortage is widely known as the data wall or token crisis.

To continue scaling model capabilities when unique human text runs out, researchers are shifting away from brute-force data accumulation. Instead, the frontier focuses on maximizing the information extracted per token and exploring alternative scaling dimensions.

Reusing and Reformulating Existing Data

If new text cannot be found, the most intuitive alternative is to read the same text multiple times. However, standard deep learning models suffer from "multi-epoch degradation"—a form of severe overfitting where repeating a dataset yields diminishing returns and harms generalization. Fuzhao Xue et al. document this phenomenon in To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis, showing that performance degrades rapidly when large models are trained over multiple epochs.

To make data repetition viable, researchers have developed frameworks to model how information accumulates across epochs. Weidong Zhou et al. introduce this paradigm in InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition (and similarly Fengze Liu et al. in LayerMix Law: Scaling Law for Large Language Models on Quality-Weighted Mixture Data with Repetition). Rather than counting raw tokens, these data-aware scaling laws measure information accumulation, allowing practitioners to predict the exact point where repeating a dataset yields diminishing returns. Crucially, Tingkai Yan et al. prove in Larger Datasets Can Be Repeated More: A Theoretical Analysis of Multi-Epoch Scaling in Linear Regression that larger baseline datasets inherently tolerate more repetitions before overfitting, providing a theoretical foundation for multi-epoch training.

Standard Scaling:  [Unique Tokens] ──> Linear Performance Gains
Data-Wall Scaling: [Repeated/Reformulated Tokens] ──> Modeled via Information Accumulation (InfoLaw)

Instead of literal repetition, a more powerful approach is data reformulation—using an auxiliary language model to rewrite noisy, low-quality web text into highly structured formats. Marco Bellagente et al. in Rephrasing natural text data with different languages and quality levels for Large Language Model pre-training and Pratyush Maini et al. in Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling demonstrate that transforming raw web scrapes into clean Q&A pairs or encyclopedic entries drastically improves data efficiency.

By altering the surface form while preserving the underlying facts, reformulation prevents the model from memorizing exact token sequences. For instance, the authors of Reformulation for Pretraining Data Augmentation show that rewriting a core dataset into diverse genres and for different target audiences successfully bypasses multi-epoch degradation entirely, effectively multiplying the utility of a fixed data budget.

Increasing Token Density via Augmentation and Synthesis

When the total volume of available tokens is capped, models must extract more signal from each individual token. This can be achieved by increasing the information density of the training data.

One prominent method is to make implicit reasoning explicit during pre-training. In Thinking Augmented Pre-training, Liang Wang et al. introduce a technique that appends automatically generated "thinking trajectories" to raw text. Instead of merely predicting the next word in a sentence, the model is trained to predict the underlying cognitive steps that lead to that sentence. This structural augmentation forces the model to learn algorithmic logic rather than surface-level correlations, boosting data efficiency by orders of magnitude.

Standard Text:      "The answer is 42."
Thinking Augmented: "The problem asks for the product of 6 and 7. 6 times 7 is 42. Therefore, the answer is 42."

Alternatively, researchers are bypassing human text altogether by using agentic frameworks to generate entirely synthetic corpora. As detailed by the authors of MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation, teams of LLM agents can be orchestrated to generate highly diverse, domain-specific datasets that rival or outperform real-world data.

However, synthetic data is not a silver bullet. Yanzhu Guo et al. warn in The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text of an "echo chamber" effect: recursively training models on synthetic text causes a severe collapse in linguistic diversity, making subsequent models increasingly repetitive and uniform.

Shifting the Scaling Frontier to Compute

If the data budget is strictly capped and cannot be expanded through repetition or synthesis, the final alternative is to scale computation along dimensions other than dataset size.

Konwoo Kim et al. explore this paradigm in Pre-training under infinite compute. They demonstrate that when data is fixed but computational resources are virtually unlimited, massive performance gains can still be unlocked. This is achieved by heavily tuning regularization, optimizing hyperparameters far beyond standard compute-optimal recommendations, and training large model ensembles that are subsequently distilled into smaller, practical architectures.

Traditional Scaling:  More Compute ──> More Data + Larger Models
Data-Constrained:     More Compute ──> Heavy Regularization + Hyperparameter Tuning + Ensembling

Another approach is to shift the model's intelligence from the pre-training phase to the inference phase. Deqian Kong et al. introduce this concept in Latent Thought Models with Variational Bayes Inference-Time Computation, which injects continuous "thought vectors" into the Transformer architecture, allowing the model to allocate extra compute per token during inference.

Expanding on this, Boyi Zeng et al. present PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space, demonstrating that scaling these continuous latent thoughts during pre-training allows smaller models to match the capabilities of architectures with twice their parameter count.

As summarized by Hanyu Lai et al. in A Survey of Post-Training Scaling in Large Language Models, the industry is undergoing a fundamental transition. As the human data wall prevents us from scaling pre-training data, the next generation of language model intelligence will be driven by post-training refinement, data-efficient architectures, and inference-time computation.

Key Foundational Work

An empirical analysis of compute-optimal large language model training (2022) — contributed 3 concept elements to this direction.
Template-free Data-to-Text Generation of Finnish Sports News (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_training_data_scarcity_50434168,
  title = {training data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/training-data-scarcity/rdr_50434168a8c44f7da5bd53c04e078cc1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_training_data_scarcity_50434168_v1,
  title = {training data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/training-data-scarcity/rdr_50434168a8c44f7da5bd53c04e078cc1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake