AI models collapse when trained on recursively generated data

Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross J. Anderson, Yarin GalNat 2024 · DOI · OpenReview

In the current generative AI boom, high-quality data is often compared to oil. However, a significant portion of the data now being produced is synthetic—the byproduct of previous AI generations. A recent study published in Nature by researchers from Oxford, Cambridge, Toronto, and Imperial College London warns of a phenomenon called "model collapse." This degenerative process occurs when AI models are trained on recursively generated data, eventually causing them to lose their representation of reality.

The Problem of Recursive Training

Large Language Models (LLMs) like GPT-4 have historically been trained on massive datasets scraped from the internet, which were almost entirely human-generated. As AI-generated content becomes ubiquitous online, a critical feedback loop emerges: GPT-$n$ is increasingly trained on the output of GPT-$(n-1)$.

Conceptual framework of model collapse showing the feedback loop from model-generated data back into training sets.

The researchers investigated whether this recursive training remains stable or leads to drift. They aimed to quantify "data pollution" and understand the mathematical reasons why models fail when they stop learning from humans and begin learning from their own previous outputs.

Methodology

The study combined theoretical proofs with empirical experiments across three model types: Gaussian Mixture Models (GMMs), Variational Autoencoders (VAEs), and Large Language Models (LLMs).

  1. Theoretical Modeling: Markov chain analysis and Wasserstein-2 distance metrics were used to prove that model collapse is an inevitable mathematical outcome of recursive sampling.
  2. LLM Fine-tuning: Researchers fine-tuned an OPT-125m model on the Wikitext2 dataset. This model then generated a new dataset used to train the subsequent generation.
  3. Experimental Variants: Two scenarios were tested: Total Replacement (training only on the previous generation's output) and Data Preservation (mixing 10% of the original human data into each synthetic training set).

The Mechanism of Collapse

Model collapse is characterized by the "vanishing tails" of a data distribution. It typically progresses through two stages:

  • Early Model Collapse: The model begins to "forget" improbable events (the tails of the distribution). Because the model is less likely to generate these rare examples, the next generation sees even fewer of them.
  • Late Model Collapse: The model converges to a narrow, low-variance point estimate that fails to represent the complexity of the original data.

The paper identifies three primary sources of error driving this decay: Statistical Approximation Error (missing rare events due to finite samples), Functional Expressivity Error (model capacity limits), and Functional Approximation Error (biases in the learning procedure).

Key Results

Empirical testing on the OPT-125m model demonstrated significant degradation across generations.

Empirical results for OPT-125m on Wikitext2. Histograms show the narrowing perplexity distribution (vanishing tails), while line plots track perplexity drift across generations for both total replacement (b) and 10% data preservation (c).
  • Perplexity Drift: In the experiments, the model's perplexity on the real test dataset rose significantly, indicating a loss of predictive power. Conversely, the perplexity of the generated data points dropped (e.g., from 34 to 28), which signaled a loss of complexity as the model became overconfident in a narrow subset of language.
  • The "Jackrabbit" Effect: Qualitatively, later generations produced repetitive, nonsensical outputs. In one instance, a model began obsessing over "black @-@ tailed jackrabbits" regardless of the prompt, illustrating a total collapse into a narrow vocabulary subset.
  • Preservation Limits: While keeping 10% of the original human data slowed the collapse, it did not prevent the eventual loss of the distribution's full breadth.

Implications and Future Work

These findings suggest a massive "first-mover advantage" for companies that trained models on the pre-2023, human-dominated internet. Newcomers scraping the web today face "polluted" data, making it harder to achieve high performance. Furthermore, model collapse poses a fairness risk: as the "tails" vanish, marginalized groups and niche perspectives—which often reside in those tails—are systematically erased from AI outputs.

Future research directions include developing robust data watermarking to filter synthetic content, community coordination to share "clean" datasets, and investigating algorithmic architectures that may be more resistant to recursive decay.

Summary updated

Abstract

Stable diffusion revolutionized image creation from descriptive text. GPT-2 (ref. 1), GPT-3(.5) (ref. 2) and GPT-4 (ref. 3) demonstrated high performance across a variety of language tasks. ChatGPT introduced such language models to the public. It is now clear that generative artificial intelligence (AI) such as large language models (LLMs) is here to stay and will substantially change the ecosystem of online text and images. Here we consider what may happen to GPT-{n} once LLMs contribute much of the text found online. We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as ‘model collapse’ and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs). We build theoretical intuition behind the phenomenon and portray its ubiquity among all learned generative models. We demonstrate that it must be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet.  Analysis shows that indiscriminately training generative artificial intelligence on real and generated content, usually done by scraping data from the Internet, can lead to a collapse in the ability of the models to generate diverse high-quality output.
BibTeX
@article{shumailov2024ai,
  author = {Ilia Shumailov and Zakhar Shumaylov and Yiren Zhao and Nicolas Papernot and Ross J. Anderson and Yarin Gal},
  title = {{AI models collapse when trained on recursively generated data}},
  journal = {Nat.},
  year = {2024},
  doi = {10.1038/s41586-024-07566-y},
}
Figure