The internet has been the training ground for modern large language models. We've scraped billions of web pages, books, code repositories, and conversations. But we're hitting a ceiling. By some estimates, we've already trained models on most of the high-quality text data available online. The obvious next question: what happens next?

The answer is synthetic data. But not the synthetic data of five years ago—generated images that looked obviously fake, or paraphrased text that added no new signal. Today, we have self-play environments where models generate reasoning traces for complex problems. We have constitutional AI systems that generate massive preference datasets for RLHF. We have techniques for generating domain-specific data that mimics real distributions without violating privacy. And emerging evidence suggests this synthetic data doesn't just supplement real data—it can outperform it for specific capabilities.

In this article, I'll walk through the frontier of synthetic data generation: why the internet data is exhausting, what failure modes arise when you train on model-generated data, and which new techniques—self-play, constitutional AI, verification-based filtering, and domain-specific synthesis—are proving effective at scale. This reflects both published research and insights from the teams building the next generation of frontier models.

The Data Wall: Why Internet Data Exhaustion Matters

The conventional narrative frames LLMs as data-hungry. Train on more data, get better models. But this relationship has two critical boundaries: quantity and quality.

The Quantity Ceiling

Estimates of high-quality English text on the internet vary, but most researchers converge on 1-5 trillion tokens. This sounds enormous until you realize:

Chinchilla scaling laws tell us that optimal model performance requires balanced scaling: compute ∝ parameters ∝ training tokens. If we can't scale training tokens further, we hit a ceiling on model size and compute efficiency.

The Quality Crisis

But quantity is only part of the story. The internet contains signal and noise in vastly unequal proportions. Web text includes:

Existing models attempt to filter this (through automated classifiers, ratio-based heuristics, quality scoring). But the signal-to-noise tradeoff is unforgiving. Aggressive filtering removes useful niche content. Lenient filtering introduces noise that hurts model quality downstream.

The Paradox

Models trained on curated, high-quality datasets (like academic textbooks) have lower loss but sometimes underperform models trained on noisy web data. This suggests diversity and coverage matter more than purity—but only up to a threshold of coherence.

The Model Collapse Problem: Training on AI-Generated Data

If we simply ask models to generate training data for themselves, we get a dangerous failure mode called model collapse. This wasn't well understood until recently, but it's now a core research question: what happens when models train on data generated by earlier versions of themselves?

How Model Collapse Works

Imagine this loop: Train model M1 on web data. Use M1 to generate synthetic data. Train model M2 on a mix of web data and M1-generated data. Use M2 to generate synthetic data. Train model M3 on a mix of web data and M2-generated data. And so on.

At each generation, the statistical distribution of the synthetic data shifts toward the center of M's learned distribution. Outliers and rare patterns get smoothed. Diversity contracts. Within a few generations, you end up with models that are increasingly generic and less capable on out-of-distribution tasks.

The mechanism: models are lossy compressors. They learn the high-probability region of the training data. When they generate new data, it's sampled from that learned distribution. Each generation sharpens the distribution (lower entropy, higher concentration). The original diversity of web data—which includes rare but important patterns like edge cases, counterexamples, and adversarial queries—gets progressively erased.

Recent papers on this (e.g., Shumailov et al., 2024) show that model collapse is not hypothetical—it happens predictably when you feed models their own outputs repeatedly. The kicker: you might not notice until performance degrades significantly, sometimes after multiple training runs.

Self-Play: How Models Generate Better Data Than Humans

Model collapse happens when you naively generate data. But what if you structure the generation process strategically?

Self-play is a paradigm where models generate reasoning traces—step-by-step solutions to problems—rather than surface-level answers. The key insight: humans are bad at generating exhaustive, detailed reasoning. Models are good at reasoning through complex problems. If you ask a model to solve a problem and show all work, you get something humans couldn't easily create manually.

Self-Play for Mathematical Reasoning

Consider training a model on mathematical problem-solving. Manually, you might create datasets of 10,000 problems with solutions. But manual solutions often skip steps, assume reader knowledge, or take shortcuts. Self-play flips this: you ask the model to generate detailed chains-of-thought for problems, then use those traces as training data.

Concretely:

  1. Start with a base model M0 (trained on web data + standard problem solutions)
  2. Generate 100,000 chain-of-thought solutions: for each problem, prompt M0 with "Solve this step-by-step, showing all reasoning"
  3. Collect the reasoning traces as synthetic data
  4. Train M1 on web data + these reasoning traces
  5. Evaluate M1 on held-out problems (not seen during generation)

Empirically, this works. Models trained on self-generated reasoning traces outperform models trained only on web data, even when total token count is identical. Why? Because the reasoning traces encode the model's own problem-solving process—a direct signal for "how to think about hard problems."

Preventing Collapse During Self-Play

But self-play isn't immune to model collapse. If you generate traces from a weak model and train on them, you lock in those weaknesses. The solution is iterative refinement with verification:

This is what models like OpenAI o1 use (though the exact architecture is proprietary). You generate multiple reasoning paths, keep only the ones that work, and train the next version on high-quality, verified traces. The verification step is crucial—it prevents collapse by filtering out models' own errors.

Constitutional AI and Preference Data Generation

Another frontier of synthetic data: generating RLHF (Reinforcement Learning from Human Feedback) preference data without human annotators. Anthropic's constitutional AI framework does this by having models critique and improve their own outputs according to a constitution—a set of explicit principles.

The Constitutional AI Loop

The process:

  1. Prompt the model: "Write a short story about..."
  2. Generate two outputs: Model produces response A and response B (e.g., A is the default, B is from a variant or instruction tweak)
  3. Critique: Ask the model: "Which response is better according to these principles: [constitution]. Explain your reasoning."
  4. Collect preferences: Use the model's critique as a training signal. Response A > Response B (or vice versa)
  5. Reward function: Train a reward model on these synthetic preferences
  6. RL loop: Use the reward model to fine-tune the base model via reinforcement learning

The magic: the model's own critiques encode human-like judgment, without hiring human raters. A single model can generate preferences for millions of examples in days, compared to months for human annotation at scale.

This has shown strong empirical results. Models fine-tuned on constitutionally-generated preferences sometimes match or exceed models tuned on human preferences, especially when the constitution aligns with human values.

Verification-Based Filtering and Decontamination

Not all synthetic data is equally useful. The frontier is moving toward active quality control: measuring the quality of generated data before training.

Verification for Code and Math

For domains with clear correctness criteria (code, mathematics, logic), you can verify generated outputs automatically:

This is higher-signal than unverified data. A 10,000-example dataset of code that all passes tests is more valuable than 1 million lines of unverified code.

Quality Scoring for Text

For open-ended text (essays, creative writing, analysis), verification is harder, but proxy metrics help:

The cutting edge: decontamination. Before training on synthetic data, remove any examples that are too similar to web data already seen (which would just be reinforcing existing patterns). Use semantic similarity (embeddings) or exact string matching to identify and filter duplicates.

Domain-Specific Synthetic Data: The Clinical Documentation Example

One of the most promising applications of synthetic data is in specialized domains where real data is expensive, scarce, or privacy-sensitive. Medical AI is a prime example.

Consider clinical documentation: medical notes written by doctors for electronic health records (EHRs). High-quality medical AI requires models trained on this data. But real clinical notes are highly protected (HIPAA in the US), and there's limited labeled data available for research and model development.

Synthetic Medical Data at Scale

My colleagues and I published research on this (arXiv:2406.06569, "Generating Clinical Documentation with LLMs"). The approach:

  1. Start with schema: Define the structure of a clinical note (chief complaint, history of present illness, physical exam findings, assessment, plan, etc.)
  2. Generate synthetic patients: Use a model to create plausible patient profiles (age, gender, medical history, presenting symptoms)
  3. Generate synthetic notes: For each synthetic patient, prompt the model to write a realistic clinical note consistent with the schema and patient history
  4. Validate: Check that notes are clinically coherent, don't contain dangerous hallucinations, and follow proper medical terminology
  5. Anonymize: Ensure no real patient information is embedded

The result: thousands of realistic clinical notes with no privacy violations, ready for training medical AI models. The synthetic notes follow real documentation patterns, include appropriate medical terminology, and cover diverse clinical presentations.

This is a template for synthetic data in other regulated domains: finance (synthetic transaction logs), law (synthetic contract clauses), manufacturing (synthetic sensor logs), etc. The pattern is always the same: structure + generation + validation.

Evaluation Frameworks: How Do You Know It Works?

The hardest question: how do you measure whether synthetic data actually helped? It's not enough to check training loss or benchmark scores. You need principled evaluation.

Held-Out Real Data

The gold standard: train on synthetic data + web data, evaluate on held-out real data that was never seen during generation or training. If the model performs well on real data, the synthetic data transferred the capability.

But this has a catch: if your synthetic data doesn't cover a specific phenomenon, the model will underperform on that phenomenon in evaluation. This is actually useful—it tells you what your synthetic generation pipeline is missing.

Diversity Metrics

Measure whether your synthetic dataset has sufficient diversity:

Model Behavior Under Distribution Shift

Train two models: one on synthetic data alone, one on synthetic + real data. Test both on out-of-distribution examples (data unlike anything in the training set). If the synthetic-only model fails gracefully and the combined model performs significantly better, the synthetic data helped calibrate behavior.

Comparison: Synthetic Data Approaches at a Glance

Approach Best For Collapse Risk Verification
Self-Play + Verification Reasoning, math, code Low (oracle-verified) Strong (automated checks)
Constitutional AI Preference data, RLHF Medium (depends on constitution) Medium (model-based critique)
Domain-Specific Generation Specialized domains (medical, legal) Medium (mitigated by schema) Medium (domain-expert validation)
Diversity-Filtered Web Scraping Broad, general knowledge Low (real data) Weak (heuristic-based)
Naive Text Generation Almost nothing (deprecated) Very High None

The Future: Recursive Improvement and Emergence

Where is this heading? Several emerging trends:

Recursive Improvement Without Distribution Shift

Models that generate their own training data, verify it, and fine-tune on it repeatedly—but with mechanisms to prevent model collapse. This creates a tightening loop: each generation of the model is slightly better than the last, enabled by synthetic data from its predecessor. If done carefully, this could unlock continuous improvement without requiring new human data.

Mixture-of-Experts Synthetic Data

Different synthetic data sources for different capabilities: one pipeline generates math data, another generates code, another generates reasoning traces. Each pipeline is optimized for its domain. Models train on a mixture, learning to route different problems to different learned skills.

Synthetic Data for Alignment

As models get more capable, ensuring they stay aligned with human values becomes harder. Synthetic data generation could play a role: generate millions of edge cases that test alignment (adversarial queries, ambiguous instructions, potential misuse scenarios), verify they're handled correctly, and fine-tune the model on high-quality alignment data. This could automate part of the alignment pipeline.

Emergence Through Synthetic Data

Perhaps most speculatively: could synthetic data drive emergence of new capabilities? If you generate data that requires reasoning skills the model doesn't yet have, train on it, and iterate, does the model gradually develop those skills? Early evidence from constitutional AI and self-play suggests yes—but the mechanisms are not yet understood.

The Bottom Line

We've hit a wall with internet-scale training data. The quantity is finite, the quality mixed. But this isn't a dead end—it's a transition. The next generation of models will be trained on designed data, not scraped data. This data will be structured, verified, diverse, and domain-specific.

The frontier isn't "make better synthetic data." It's "make synthetic data that's specifically optimized for the capability you want to teach." Self-play for reasoning. Constitutional AI for values alignment. Domain-specific generation for specialized applications. Verification for correctness. The complexity is moving from "how much data" to "what kind of data, in what order, with what structure."

For practitioners building models today: assume synthetic data will be 30-50% of your training mix within 2-3 years. Start experimenting with verification-based pipelines and constitutional AI now. The sooner you understand how to generate, verify, and filter synthetic data, the earlier you'll have access to models trained on it.