Who Pays for the Knowledge Models Copy? Model Collapse as Rogers' Paradox

Nam H. LearXiv 2026 · arXiv

TL;DR

Model collapse—the degradation of generative models trained on synthetic data—is not just a statistical anomaly, but a manifestation of Rogers’ Paradox from cultural evolution. This paper demonstrates that AI reliability is ultimately "paid for" by human effort; without "critical learners" who verify facts against the real world, models inevitably go stale, especially as the world changes.

What Problem Does This Paper Address?

As generative AI becomes ubiquitous, the internet is increasingly flooded with model-generated text. When the next generation of models is trained on this "synthetic" data, they begin to forget rare information and converge on a narrow, repetitive distribution—a phenomenon known as model collapse.

Existing theories often treat the supply of "fresh" human data as an infinite, exogenous resource. However, in reality, the supply is endogenous: people use AI to write the very data that will train future models. Furthermore, most analyses assume a static world, ignoring that information has a "shelf life." The author argues that we need a theory that accounts for the cost of information and the dynamics of a changing world.

Methods

The author identifies model collapse with Rogers’ Paradox, a 1988 theory from cultural evolution. The paradox states that a population that relies on cheap copying (social learning) ends up no better off than one that learns for itself, because copied information eventually goes stale.

The paper tests this identification through:

  1. Quantitative Modeling: Applying a simple recursive equation from Rogers’ model to predict rare-word retention in LLMs.
  2. Lineage Experiments: Training successive generations of models (word trigrams, character-level transformers, and the 125M-parameter OPT model) on varying mixtures of fresh and synthetic data.
  3. Moving World Simulations: Introducing a "sliding window" of text to simulate a world where facts change over time.
  4. Agent-Based Modeling: Simulating a human-AI ecosystem where agents choose between "asking the model" (cheap) or "finding out for themselves" (costly).

Datasets

  • Project Gutenberg: Specifically Darwin’s On the Origin of Species and Melville’s Moby-Dick for trigram and character-transformer experiments.
  • Wikitext-2: Used for fine-tuning the OPT-125m model.
  • Synthetic Data: Generated by each model generation to train the subsequent one.

Key Insights and Ideas

  • The Staleness Parameter ($u$): In this framework, $u$ represents the rate at which the world changes. The paper shows that even in a "fixed" world, $u$ captures the probability that a rare item is simply not sampled by the model, leading to its eventual disappearance.
  • Pretrained Weights as a "Knowledge Reserve": Models like OPT-125m that are fine-tuned afresh each generation are partially insulated from collapse. The pretrained weights act as a "fixed archive" of knowledge that never enters the decaying copying chain.
  • The Endogenous Trap: If AI makes it cheaper to "ask" than to "find out," people stop doing the hard work of learning. At equilibrium, the model’s accuracy is pinned exactly to the cost of finding things out. If it costs 30% of your effort to verify a fact, the model will settle at roughly 70% accuracy.
  • Filters as Social Strategies:
    • Conformist Filters: Selecting for "plausible" or "likely" text (common in RLHF or perplexity filtering) actually accelerates collapse by eroding variation faster than random sampling.
  • Critical Learning: Only "critical learners"—users who check the model against the world and relearn when it's wrong—can break the paradox and keep the ecosystem's knowledge from collapsing.

Key Results

  • Predictive Power: Rogers’ simple recursion predicted rare-word coverage in LLMs to within ~3 percentage points across different fresh-data mixtures.
  • The Price of Change: In a changing world, the amount of fresh data needed to maintain 80% knowledge coverage rose from 30% to 45% as the rate of world-change increased.
Figure 1: Lineages of language models trained on their own output in a fixed world (word trigram models, 12 generations, mean of 5 seeds). (a, b) Rare-word coverage relative to generation 0. Thin dashed lines: Rogers’ chain (Eq. 1) with u fitted to the all-synthetic condition only and applied, unchanged, to the fresh-data conditions. (c) Every condition at generation 12, on coverage and on the growth of perplexity on held-out human text. Filters are discussed in Section 7.
  • Filter Failure: In transformer models, keeping only the "most likely" half of samples (a conformist strategy) was significantly worse than keeping a random half, leading to faster error growth.
Figure 2: Recursive training of a small character-level transformer (Darwin; 6 generations; mean ± s.d. over 3 seeds). (a) Bits per character on held-out human text. (b) Rare-word coverage of the training corpus, relative to generation 0. Fresh data stop the decline; keeping the most likely half of the samples accelerates it relative to keeping a random half.

Limitations and Future Work

  • Scale: The largest model tested was 125 million parameters. While the patterns held across trigrams and transformers, testing at the scale of GPT-4 or Llama-3 remains an "important open test."
  • Simplified Drift: The "moving world" was simulated as a sliding window of existing text, which may not perfectly capture the arrival of entirely new, unprecedented facts.
  • Homogeneous Costs: The ecosystem model assumes everyone faces the same cost to learn or verify. Future work could explore heterogeneous costs, market dynamics for fresh data, and competition between multiple models.

Summary updated

Abstract

Generative models trained on the output of earlier models lose knowledge across generations, rare knowledge first. The remedies that work all bring in information from outside the loop, and the theory of model collapse treats that supply as given: a fixed archive, drawn from a fixed distribution. We argue that the supply is the problem, and that cultural evolution has the theory of it. We read model collapse as Rogers' paradox: a population that comes to rely on copying ends up no better off than one that never copied, because copies go stale unless someone keeps learning for themselves. We test the identification in three steps. First, Rogers' model, fitted only to fully synthetic training, predicts to within a few percentage points how much rare knowledge language models, from word trigrams to a pretrained transformer, retain under other mixtures of fresh data. Second, when the world itself changes, a lineage that copies its own past needs more fresh data to keep up the faster the world changes, as Rogers' model says it must and as analyses that assume a fixed target cannot; a lineage that restarts every generation from fixed pretrained weights is largely insulated, because those weights are knowledge that is never copied. Third, we close the loop: when the people who write a model's training data can ask the model instead of finding things out, the supply of fresh data becomes endogenous. At equilibrium, the model is exactly as reliable as it is costly for people to find out for themselves, the population is no better off than without the model, and every increase in the share of model-written text must be paid for by more people learning for themselves. Users who check what they are told and relearn when it is wrong break the paradox. Filters that select plausible text are conformist and accelerate collapse; only checks that carry new information act as critical learning.
BibTeX
@misc{le2026who,
  author = {Nam H. Le},
  title = {{Who Pays for the Knowledge Models Copy? Model Collapse as Rogers' Paradox}},
  year = {2026},
  eprint = {2611.06022},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06022},
}
Figure