AI Models Collapse When Trained on Recursively Generated Data (Model Collapse)¶

Authors: Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, Yarin Gal (University of Oxford, University of Cambridge, Imperial College London, University of Toronto and Vector Institute, University of Edinburgh) Published: July 2024 (Nature 631, 755-759); earlier preprint May 2023 as "The Curse of Recursion" Paper: nature.com/articles/s41586-024-07566-y

Why This Paper Matters¶

Language models are trained on text from the web. Increasingly, the web contains text written by language models. What happens when the next generation of models learns from the output of the last?

This paper gave that worry a name and a mechanism: model collapse. When models are trained recursively on data generated by earlier models, they progressively forget the rare parts of the original distribution - the tails - and eventually converge on a narrow, distorted version of it. The authors demonstrated the effect in language models, variational autoencoders and simple Gaussian mixture models, and argued it is a general statistical phenomenon.

A clear, memorable result published in Nature, which pushed "synthetic data" into mainstream discussion of AI's future.
A mechanism, not just an observation: sampling errors compound across generations, and rare events are the first to go.
A data-provenance argument: genuine human data - and knowing which data is human - becomes more valuable over time.
A contested conclusion. Follow-up work showed that collapse depends heavily on the setup, and in particular that keeping real data alongside synthetic data largely prevents it.

The insight: every model is an imperfect copy of its training distribution, and it under-samples rare events. Train the next model on those samples and the rare events become rarer still. Repeat, and they disappear.

The Mechanism¶

The authors distinguish two stages: - Early model collapse: the model loses information about the tails of the distribution - unusual phrasings, rare facts, minority cases. - Late model collapse: the model converges to a distribution with little resemblance to the original and much less variance.

And three compounding sources of error: 1. Statistical approximation error - the primary cause. Any finite sample misses some rare events; they cannot be learned if they never appear. 2. Functional expressivity error - the model family cannot represent the true distribution exactly. 3. Functional approximation error - training procedures do not find the best model even within that family.

Generation 0: trained on human data         [ wide distribution, long tails ]
Generation 1: trained on gen-0 samples       [ tails thinner ]
Generation 2: trained on gen-1 samples       [ tails mostly gone ]
   ...
Generation n:                                [ narrow peak, low variance ]

The Experiment¶

Model: OPT-125m, fine-tuned on the wikitext2 dataset. The baseline went from about 115 perplexity zero-shot to about 34 after fine-tuning on real data.
Recursive loop: each generation's model generated a new training set (using 5-way beam search in 64-token blocks), and the next generation was fine-tuned on it. Five runs per setting.
Two regimes:
No original data kept, 5 epochs per generation: perplexity on the original test data degraded substantially across generations (the paper reports degradation of roughly 20 to 28 perplexity points).
10% of the original data preserved in each generation, 10 epochs: only minor degradation.

The paper's most-quoted example follows one prompt about medieval church architecture. By generation 9, the model's continuation had drifted into a list of jackrabbits with different coloured tails.

Key Claims and Implications¶

Collapse appears across model types, suggesting a general statistical cause rather than a quirk of language models.
First-mover advantage: models trained before the web filled with AI-generated text had access to cleaner human data.
Provenance matters: the ability to distinguish human-generated from model-generated data at scale may become important for training future models.
Preserve the tails: low-probability events matter for fairness and for understanding complex systems, and they are exactly what recursive training loses.

The Counter-Evidence¶

The Nature paper's strongest version - replacing each generation's data entirely with synthetic data - is not how practitioners use synthetic data. Several follow-ups examined more realistic setups:

Accumulate, do not replace. Gerstgrasser et al. (2024, "Is Model Collapse Inevitable?", arXiv 2404.01413) showed that when synthetic data is added to real data rather than replacing it, collapse is avoided, and in a linear-model analysis test error stays bounded no matter how many generations run.
Verify synthetic data. Feng et al. (2024, arXiv 2406.07515) found that filtering synthetic data with a verifier prevents degradation and can support scaling.
Definitions matter. Schaeffer et al. (2025, "Position: Model Collapse Does Not Mean What You Think", arXiv 2503.03150) catalogued eight conflicting definitions of model collapse in the literature and argued many collapse scenarios are readily avoidable in practice.
But small amounts can still hurt. Dohmatob et al. (2024, "Strong Model Collapse", arXiv 2410.04840) argued that even a small fraction of synthetic data - on the order of 1% - can harm how models scale in some theoretical settings.

Meanwhile, deliberately generated synthetic data has become central to training strong models: instruction data (Self-Instruct), textbook-style pretraining data (phi-1), distillation (Knowledge Distillation) and self-generated reasoning traces (STaR). The difference is curation, verification and mixing with real data.

Why This Was Revolutionary¶

It named a risk everyone could understand, and moved it from speculation to evidence.
It put data provenance on the agenda for researchers and policymakers.
It forced precision. The debate that followed clarified which uses of synthetic data are risky.

Key Takeaways for Practitioners¶

Never fully replace real data with synthetic data. Keep a stream of genuine data in every training round.
Filter and verify synthetic data. Correctness checks, quality classifiers and deduplication all help.
Monitor the tails. Track rare-case performance, not just averages.
Track provenance. Know which of your training data came from models.

Limitations & Future Directions¶

Small models and specific setups. OPT-125m fine-tuning is far from frontier pretraining.
The replace-everything regime is a worst case, not common practice.
Beam search generation reduces diversity more than sampling would, likely exaggerating the effect.
A 2025 correction was published to the Nature article.
The real-world question - how much AI-generated text on the web affects future models trained on crawls - is still being measured.

Citation¶

@article{shumailov2024collapse,
  title={AI models collapse when trained on recursively generated data},
  author={Shumailov, Ilia and Shumaylov, Zakhar and Zhao, Yiren and Papernot, Nicolas and Anderson, Ross and Gal, Yarin},
  journal={Nature},
  volume={631},
  pages={755--759},
  year={2024}
}