The Curse of Recursion: Training on Generated Data Makes Models Forget

Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, Ross J. Anderson

TL;DR

As large language models (LLMs) become more prevalent, they are increasingly used to generate data that trains future models. This paper introduces "Model Collapse," a phenomenon where models trained on their own generated data irreversibly forget aspects of the original data distribution, particularly rare events. This recursive self-poisoning poses a significant threat to the long-term utility and diversity of AI models.

What Problem Does This Paper Address?

The paper addresses a critical, emerging problem in the age of powerful generative AI: what happens when the data used to train future AI models is largely generated by previous AI models? As LLMs like GPT-4 become capable of producing human-quality text and images, the internet will become increasingly populated with AI-generated content. If future models are trained on this AI-generated data, a feedback loop could emerge where models progressively "forget" the nuances and diversity of the original, human-generated data. This "model collapse" could lead to models that are less capable, less robust, and potentially biased, as they lose the ability to represent rare but important events.

Methods

The authors introduce and analyze the phenomenon of "Model Collapse" through a combination of theoretical modeling and empirical evaluation.

  • Theoretical Framework: They develop mathematical models to explain how errors accumulate over generations of training on generated data. This involves analyzing statistical approximation errors (due to finite sampling) and functional approximation errors (due to model limitations). They use concepts like Wasserstein distance and absorbing Markov chains to formalize the degradation process.
  • Empirical Evaluation: They demonstrate model collapse across various generative model types:
  • Gaussian Mixture Models (GMMs): Showed how GMMs trained on their own generated data converge to a state with very little variance, losing information about the original distribution.
  • Variational Autoencoders (VAEs): Demonstrated that VAEs trained on generated data entangle different modes of the original distribution, leading to a more unimodal latent space.
  • Large Language Models (LLMs): Investigated fine-tuning scenarios with models like OPT-125m, showing that even with some original data preserved, performance degrades over generations, and models start producing more probable, less diverse sequences.

Datasets

The paper uses existing datasets for evaluation:

  • Wikitext2: Used for fine-tuning and evaluating language models.
  • Artificially-generated Gaussians: Used for evaluating Gaussian Mixture Models.
  • Not specified for VAEs: The paper mentions generating latents and then using a decoder to generate data, but the original data source for the VAEs is not explicitly detailed.

Key Insights and Ideas

  • Model Collapse is Ubiquitous: The phenomenon is not limited to a specific model architecture but affects a wide range of generative models, including GMMs, VAEs, and LLMs.
  • Tails Disappear First: The most vulnerable parts of the data distribution, representing rare events or "tails," are the first to be forgotten. This is critical because these rare events often carry significant information or represent important edge cases.
  • Recursive Self-Poisoning: The core mechanism is a feedback loop where models trained on their own outputs progressively degrade their understanding of the original data distribution. This is akin to a form of "data poisoning" but self-inflicted.
  • Need for Original Data: Preserving access to genuine, human-generated data is crucial to counteract model collapse and maintain model performance and diversity.
  • "First Mover Advantage": Early adopters of LLMs trained on vast, clean datasets might have a temporary advantage, but this advantage is threatened by the long-term effects of model collapse if not addressed.

Key Results

  • GMMs: Within 50 iterations of re-sampling, GMMs showed significant misperception of the underlying distribution, converging to a point estimate with very little variance by iteration 2000.
  • VAEs: Generated images from later generations of VAEs appeared as a mix of all original digits, losing distinct class representations and shifting towards the mean in the latent space.
  • LLMs: Fine-tuning OPT-125m models on generated data led to performance degradation (20-28 perplexity points increase) even when some original data was preserved. Models tended to produce more probable sequences and introduced their own improbable sequences (errors).
  • Theoretical Bounds: Mathematical analysis showed that the risk (distance from the true distribution) diverges over generations unless sampling rates increase superlinearly, which is impractical.

Limitations and Future Work

  • Computational Cost: The authors note that training LLMs is computationally expensive, limiting the scope of their experiments. They opted for a proof-of-concept rather than extensive LLM training.
  • Provenance of Data: A significant challenge is tracking the provenance of data generated by LLMs at scale, making it difficult to distinguish human-generated content from AI-generated content.
  • Mitigation Strategies: The paper strongly suggests the need for access to genuine human-generated data. Future work could explore methods for identifying and preserving such data, or developing techniques to actively mitigate model collapse during training. Community-wide coordination to ensure data provenance is also proposed.

Summary updated

Abstract

Stable Diffusion revolutionised image creation from descriptive text. GPT-2, GPT-3(.5) and GPT-4 demonstrated astonishing performance across a variety of language tasks. ChatGPT introduced such language models to the general public. It is now clear that large language models (LLMs) are here to stay, and will bring about drastic change in the whole ecosystem of online text and images. In this paper we consider what the future might hold. What will happen to GPT-{n} once LLMs contribute much of the language found online? We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear. We refer to this effect as Model Collapse and show that it can occur in Variational Autoencoders, Gaussian Mixture Models and LLMs. We build theoretical intuition behind the phenomenon and portray its ubiquity amongst all learned generative models. We demonstrate that it has to be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of content generated by LLMs in data crawled from the Internet.
BibTeX
@misc{shumailov2023curse,
  author = {Ilia Shumailov and Zakhar Shumaylov and Yiren Zhao and Yarin Gal and Nicolas Papernot and Ross J. Anderson},
  title = {{The Curse of Recursion: Training on Generated Data Makes Models Forget}},
  year = {2023},
  eprint = {2305.17493},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.17493},
}
Figure