Skip to content

Synthetic Data and Model Collapse

In one line: Training models on model-generated data is now standard practice and often works very well - when it is filtered, verified, or mixed with real data - but recursively training on unfiltered model output, in place of real data, degrades models over generations. Last reviewed: 2026-09-30

The short version

  • "Synthetic data" means training data produced by a model rather than written or recorded by people: instruction-response pairs, textbook-style explanations, reasoning traces, code with tests.
  • It demonstrably helps in the right setting. Distillation from a large teacher, self-generated instructions, "textbook-quality" synthetic corpora (phi) and reinforcement learning on checked answers all rely on it. Nvidia reported that over 98% of the alignment data for Nemotron-4 340B (2024) was synthetic.
  • It demonstrably hurts in another setting. A 2024 Nature paper showed that when each model generation trains only on the previous generation's output, the rare "tails" of the data distribution vanish first and the model eventually converges on bland, distorted output. This is model collapse.
  • The difference is mostly about three things: whether real data is kept (accumulated) or replaced, whether outputs are filtered or verified, and whether new information enters the loop.
  • The open question is how much this matters as the web itself fills with AI-generated text, and how far synthetic data can substitute for the finite stock of human data.

The mental model: a photocopy of a photocopy, or a teacher with an answer key

   COLLAPSE LOOP (replace)                     USEFUL LOOP (filter / verify / accumulate)

   real data                                    real data --------------------+
      |                                             |                         |
   model 1 --> samples                           model 1 --> samples          |
                 |                                             |              |
              model 2 --> samples                        FILTER / VERIFY      |
                            |                          (tests, rewards,       |
                         model 3 ...                    stronger teacher)     |
                                                               |              |
   each copy loses rare details;                        kept samples + real data
   errors compound                                             |
                                                          model 2 (better)

A photocopy of a photocopy loses detail each round, because copying only removes information. Synthetic data works when something adds information at each round: a stronger teacher model, a test suite that checks the code, a verifier that checks the maths answer, a human rating, or the original real data kept in the mix.

When training on model outputs helps

Distillation: learning from a stronger teacher

Hinton, Vinyals and Dean's Distilling the Knowledge in a Neural Network (2015) showed a small "student" model can learn more from a large "teacher's" full output probabilities than from hard labels alone. The modern LLM version trains a small model on text a large model generates.

A prominent example: DeepSeek released six distilled reasoning models (Qwen2.5-based at 1.5B, 7B, 14B and 32B parameters; Llama-based at 8B and 70B), fine-tuned on 800,000 samples curated with DeepSeek-R1. DeepSeek reported that the distilled 32B Qwen model outperformed OpenAI's o1-mini on several benchmarks. Information flows downhill from a stronger model, so there is no collapse; the ceiling is the teacher. See Quantization and distillation.

Self-instruct: bootstrapping instruction data

Self-Instruct (2022) had a model generate new instructions and responses from a small seed set, filter out low-quality and near-duplicate ones, and fine-tune on the result. It showed instruction-following could be taught without large human-written datasets, and inspired many open instruction datasets. The filtering step matters: it removes the worst of the model's output before it is trained on.

Textbook-quality synthetic corpora: phi

Textbooks Are All You Need (phi-1, June 2023) trained a 1.3-billion-parameter code model on about 7 billion tokens: 6 billion of filtered "textbook quality" web data plus 1 billion tokens of synthetic textbooks and exercises generated by GPT-3.5. It scored 50.6% on the HumanEval coding benchmark, strong for its size at the time. The later phi-4 (14 billion parameters, December 2024) used synthetic data throughout training, and Microsoft reported it "substantially surpasses its teacher model" on STEM-focused question answering - evidence that careful synthetic data can do more than copy the teacher.

Skeptics of the phi line have questioned how well its benchmark results reflect general ability versus closeness to the benchmark's style, a concern discussed in Contamination and saturation.

Self-generated reasoning with verification: STaR and RL

STaR (2022) had a model write step-by-step rationales, kept only those that reached the correct answer, and fine-tuned on them, repeatedly. The model trains on its own output, but only output that passed a check, so each round adds information.

Reinforcement learning with verifiable rewards (RLVR), which powers reasoning models like DeepSeek-R1, takes this further: the model generates many attempts, a checker scores them (does the code pass tests, is the maths answer right), and training favours the successful ones. The verifier is an external source of truth, which is exactly what the collapse loop lacks.

Alignment data at scale

The Nemotron-4 340B technical report (June 2024) said over 98% of the data in its alignment process was synthetically generated. Synthetic preference data and model-written critiques (as in Constitutional AI) are now routine parts of post-training.

When it hurts: model collapse

The Nature paper

Shumailov, Shumaylov, Zhao, Papernot, Anderson and Gal first posted "The Curse of Recursion" in May 2023; a version was published in Nature in July 2024 as AI models collapse when trained on recursively generated data. They trained successive generations of models (Gaussian mixtures, variational autoencoders and a fine-tuned language model), each on data produced by the previous generation. They found:

  • Early collapse: the tails of the distribution disappear first. Rare words, unusual facts and minority styles are under-sampled, so the next model sees fewer of them, and so on.
  • Late collapse: the distribution narrows and drifts until output bears little resemblance to the original data, with low variance and compounding errors.
  • The effect comes from ordinary statistical sampling error and model approximation error, so it is expected to be widespread, not a quirk of one architecture.

Their warning: as AI-generated content spreads across the web, future models trained on scraped data risk this loop, which makes genuine human-generated data, and knowing where data came from, increasingly valuable.

"Self-Consuming Generative Models Go MAD" (Alemohammad et al., July 2023) found the same pattern in image generators and named it Model Autophagy Disorder: "without enough fresh real data in each generation", quality or diversity progressively decreases. Sampling tricks that favour quality (like lower temperature) speed up the loss of diversity.

The strongest counter-argument: accumulate, don't replace

Gerstgrasser, Schaeffer and colleagues ("Is Model Collapse Inevitable?", April 2024) argued the collapse setup is unrealistic. In the main collapse setups, each generation is trained mostly or entirely on the previous generation's output, replacing the earlier data. In practice, data accumulates: new synthetic text is added to the web alongside existing human text, which does not disappear. Across language models, diffusion models and VAEs they found:

  • Replace: performance degrades generation after generation, confirming collapse.
  • Accumulate real plus all synthetic generations: collapse is avoided, and they proved test error stays bounded in a simplified setting.

The reply from the other side is not that accumulation is irrelevant but that the rare tails can still thin out as synthetic data grows as a share of the mix, that labs cannot always tell human from machine text in scraped data, and that "bounded error" is not the same as "no harm".

Putting the two sides together

Factor Pushes toward benefit Pushes toward collapse
Source of new information Stronger teacher, verifier, tests, human feedback None - model learns only from itself
Real data Kept and accumulated Replaced
Filtering Aggressive quality and correctness filters Raw, unfiltered samples
Diversity Deliberately varied prompts and topics Low-temperature, repetitive sampling
Target A defined skill (maths, code, instruction following) The whole open-ended distribution of human text

Most practitioners' view is roughly: synthetic data is a powerful tool for teaching specific, checkable skills and for transferring ability from large models to small ones, and a risky substitute for real data as a general-purpose pretraining source. How far the second use can go is the unresolved part.

Why this matters for scaling

Public human text is finite; Epoch AI projects frontier training will exhaust the high-quality stock between 2026 and 2032. Synthetic data is one of the main proposed ways past that limit. If collapse is avoidable with accumulation and verification, synthetic data extends scaling; if tails erode anyway, it may improve narrow skills while slowly narrowing models' knowledge of the long tail of human writing. See Is scaling hitting a wall?

What to watch

  • Data provenance and watermarking, including the EU AI Act's requirement from August 2026 that generative systems mark synthetic content in machine-readable form, which could make it easier to filter AI text out of scraped training data. See The EU AI Act.
  • Evidence on long-tail knowledge in models trained on heavily synthetic mixtures: rare facts and low-resource languages are the first things collapse would erode.
  • Verifiers for open-ended tasks: RL-style synthetic data works where answers can be checked; extending it to writing, research and judgment depends on better automatic evaluation.
  • Distillation from closed models, which raises terms-of-service and competition questions as well as technical ones.

Sources

  • Shumailov et al., "AI models collapse when trained on recursively generated data", Nature 631 (published 24 Jul 2024): https://www.nature.com/articles/s41586-024-07566-y ; preprint "The Curse of Recursion" (arXiv 2305.17493, May 2023): https://arxiv.org/abs/2305.17493
  • Gerstgrasser et al., "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data" (arXiv 2404.01413, Apr 2024): https://arxiv.org/abs/2404.01413
  • Alemohammad et al., "Self-Consuming Generative Models Go MAD" (arXiv 2307.01850, Jul 2023): https://arxiv.org/abs/2307.01850
  • Gunasekar et al., "Textbooks Are All You Need" (arXiv 2306.11644, Jun 2023): https://arxiv.org/abs/2306.11644
  • Abdin et al., "Phi-4 Technical Report" (arXiv 2412.08905, Dec 2024): https://arxiv.org/abs/2412.08905
  • Nvidia, "Nemotron-4 340B Technical Report" (arXiv 2406.11704, Jun 2024): https://arxiv.org/abs/2406.11704
  • DeepSeek-R1 model card (distilled models, 800k samples): https://huggingface.co/deepseek-ai/DeepSeek-R1 ; paper (arXiv 2501.12948): https://arxiv.org/abs/2501.12948
  • Epoch AI, "Will we run out of data?" (June 2024): https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-data
  • Hinton, Vinyals and Dean (2015), Wang et al. (Self-Instruct, 2022), Zelikman et al. (STaR, 2022): see the linked summaries in this repo.