Public code for Model Collapse

Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, Ross Anderson

Abstract: Stable Diffusion revolutionised image creation from descriptive text. GPT-2, GPT-3(.5) and GPT-4 demonstrated astonishing performance across a variety of language tasks. ChatGPT introduced such language models to the general public. It is now clear that large language models (LLMs) are here to stay, and will bring about drastic change in the whole ecosystem of online text and images. In this paper we consider what the future might hold. What will happen to GPT-{n} once LLMs contribute much of the language found online? We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear. We refer to this effect as Model Collapse and show that it can occur in Variational Autoencoders, Gaussian Mixture Models and LLMs. We build theoretical intuition behind the phenomenon and portray its ubiquity amongst all learned generative models. We demonstrate that it has to be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of content generated by LLMs in data crawled from the Internet.

1. Introduction

We are currently in a "gold rush" for Large Language Models (LLMs), with models like GPT-4 trained on massive scrapes of the internet. However, as the digital ecosystem becomes increasingly populated with AI-generated text and images, a looming systemic risk emerges: Model Collapse. This paper investigates the consequences of models "eating their own tail"—training on the output of their predecessors. Unlike catastrophic forgetting or data poisoning, Model Collapse is an inevitable degenerative process where errors in statistical sampling and model approximation compound over generations, leading to an irreversible loss of information about the true underlying data distribution.

2. The Mechanism of Model Collapse

Model collapse is driven by two primary sources of error that manifest during recursive training. First, Statistical Approximation Error occurs due to finite sampling; if rare events (the "tails" of a distribution) are not captured in one generation's sample, the subsequent model will never learn of their existence. Second, Functional Approximation Error arises from the inherent limitations of neural networks, which may fail to capture the full complexity of the data or assign probability to "nonsense" regions.

Conceptual cycle of Model Collapse showing the feedback loop between data and models, where probable events are over-estimated and tails shrink over time.

As illustrated in the figure above, the process creates a feedback loop. Probable events begin to "poison reality" by becoming over-represented, while improbable events are progressively filtered out. This results in a "random walk" where model parameters drift further from the true distribution over time.

3. Methodology

The researchers evaluated this phenomenon across three distinct model classes:

  1. Gaussian Mixture Models (GMMs): To provide a low-dimensional analysis of distribution shifts.
  2. Variational Autoencoders (VAEs): Tested on the MNIST dataset to visualize latent representation evolution.
  3. Large Language Models (LLMs): Specifically fine-tuning the OPT-125m model on the WikiText-2 dataset.

The core methodology involved recursive fitting-sampling: training Model $n$ on data from Model $n-1$, generating a new dataset, and repeating the process while measuring the divergence from the original human-generated ground truth.

4. Results and Discussion

The experiments revealed a consistent pattern of decay. In "early" model collapse, the model loses information about rare events at the edges of the distribution. In "late" collapse, the distribution converges to a narrow mean with near-zero variance.

Empirical evidence of distribution collapse, where resampled data loses the tails of the original distributions and converges toward high-probability peaks.

Key findings include:

  • LLM Performance Decay: Fine-tuning OPT-125m on its own generations caused perplexity to rise from ~20 to ~28. Qualitatively, models began producing repetitive gibberish, such as an obsession with "jackrabbits" regardless of the prompt context.
  • VAE Degradation: By the 20th generation, VAE reconstructions of MNIST digits became indistinguishable, blurring different numbers into a single, unimodal average shape.
  • Data Preservation: While keeping 10% of original human data in the training set slowed the collapse, it did not prevent the eventual degradation of the model's perception of reality.

5. Conclusion

Model collapse highlights a significant "first mover advantage" for models trained on the pre-AI internet. As the web becomes polluted with synthetic content, the value of original, human-produced data—such as books and genuine forum interactions—will skyrocket. To sustain the benefits of generative AI, future research must prioritize data provenance, watermarking, and the development of training techniques that are robust to the compounding errors of recursive data loops.

Summary updated

Abstract

Stable Diffusion revolutionised image creation from descriptive text. GPT-2, GPT-3(.5) and GPT-4 demonstrated astonishing performance across a variety of language tasks. ChatGPT introduced such language models to the general public. It is now clear that large language models (LLMs) are here to stay, and will bring about drastic change in the whole ecosystem of online text and images. In this paper we consider what the future might hold. What will happen to GPT-{n} once LLMs contribute much of the language found online? We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear. We refer to this effect as Model Collapse and show that it can occur in Variational Autoencoders, Gaussian Mixture Models and LLMs. We build theoretical intuition behind the phenomenon and portray its ubiquity amongst all learned generative models. We demonstrate that it has to be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of content generated by LLMs in data crawled from the Internet.
BibTeX
@misc{shumailov2023public,
  author = {Ilia Shumailov and Zakhar Shumaylov and Yiren Zhao and Yarin Gal and Nicolas Papernot and Ross Anderson},
  title = {{Public code for Model Collapse}},
  year = {2023},
  eprint = {2305.17493},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.17493},
}
Figure