Data is the food our large language models grow on, and lately the pantry is running low. The high-quality web data that made these models great has largely been eaten, and there is increasingly little fresh content readily available to harvest. Researchers call this the data wall.

Since large language models are really good at generating coherent-sounding outputs, an obvious idea is to have these models make their own meals: let one model generate the outputs that train the next. Picturing a model making food should conjure images of plastic sushi in a restaurant window; it might look like food, but you would not want to eat it. A model raised on a steady diet of other models' generated outputs ultimately becomes malnourished and starts to drift from reality, loses variety, and degrades a little more with each generation, a failure called model collapse.

The better idea is not to generate fake food, but to make more dishes from the available real ingredients. Think of a dozen eggs. The eggs become a fluffy omelet, a silky custard, a frittata, a meringue, or a poached egg over toast. One ingredient, many dishes, each one expressing the richness and texture of the egg, and every one of them still real food that one would want to eat. That’s what the leading approach to synthetic data does: it takes real documents and then re-prepares them into forms that a model can learn from more easily, without leaving the essence and the flavor of the real ingredients behind.

Two approaches to synthetic data

There are two approaches to consider for generating synthetic data.

The first approach generates new data from scratch. A language model writes new material from a prompt, often in a clean, textbook style, as in Microsoft's Phi models, the TinyStories work, and the Cosmopedia dataset. This method gives a development team control over format, but the output can only contain what the generator model already knows, so it carries forward that model's blind spots and biases. Pushed far enough, this is the path that leads to model collapse.

The second approach, pioneered at scale by Datology and adopted by open and foundation model builders, rephrases existing documents. Instead of inventing knowledge, models rewrite real source material into more learnable forms. A single passage might become a set of question-and-answer pairs, a podcast interview, or a plain-language explanation. Our research has shown that this variety is a feature: it keeps a model learning where any single recipe would quickly plateau. It is the same move as the eggs. Same ingredients, different dishes, all of it still real food. Because the output stays anchored to real documents, the facts and the real-world distribution survive, and the recursive drift that causes collapse never gets started. The research bears this out: collapse results from cutting ties to real-world data, and keeping synthetic data grounded in real-world material prevents it.

It works on more than the web

The technique grew up on web text, in methods like Web Rephrase Augmented Pre-training (WRAP), but the same approach works on any corpus, including code and math, and it is often most valuable on proprietary data, where the material is rich but limited in volume. Working with Thomson Reuters, the team at Datology rephrased and upsampled their proprietary legal documents into synthetic training material. A finite but irreplaceable body of legal expertise became far more learnable than the raw files alone, and midtraining on the curated data yielded a 5 percentage point gain over the previous model on legal benchmarks, and doubled the improvements from post-training, all at less than 1% of the original pretraining budget.

Synthetic across all stages of training

Synthetic data is useful at every stage of training. In pretraining, the job is breadth: working with Arcee AI, we paired more than 10 trillion synthetic tokens with 10 trillion curated web tokens to build a 20-trillion-token corpus for their Trinity models. In mid-training, depth in a specific domain becomes more important, which is exactly where the Thomson Reuters work fits, and where the most useful data tends to be proprietary or rare. In post-training, synthetic data supplies the instruction, preference, and reasoning examples that teach a model how to be useful, work that is slow and expensive to do by hand.

An evolving field

Synthetic data using rephrasing techniques is now seen as the standard and is used in models from open weights to frontier labs. In BeyondWeb, the team at Datology showed that synthetic data generated using these techniques at a trillion-token scale beat the strongest public synthetic datasets of the time, and our results have only improved since then.

For as long as machine learning has existed, data has been treated like a natural resource: something you gather and consume without much thought, but the game has changed at the volumes required for today’s AI models. Synthetic data that maximizes diversity but stays grounded in real documents helps expand the menu so our models eat a veritable rainbow variety of healthy foods, resulting in models with the right mix of inputs and much stronger outputs in the form of more accurate and more concise responses that are absolutely critical for today’s hungry reasoning agents.

Frequently Asked Questions about Synthetic Data

What is synthetic data in AI training?

Synthetic data is training data produced by a language model rather than scraped from the web or generated by a business process. In modern AI training, the best approach uses a model to write, rewrite, or restructure text so that a new model can learn from it more efficiently. It is not fabricated or random content; the most effective forms stay anchored to real source material.

What is the "data wall"?

The data wall is the point at which adding more web data no longer improves AI models, because the learning signal in existing text on the open web has largely already been exhausted. Synthetic data is one of the AI field's main responses to the data wall.

Why do AI models need synthetic data?

Data is what large language models learn from, and the supply of high-quality human-written text is running low. Synthetic data lets models get more learning from existing data, fill gaps the open web never covered, and put proprietary or rare material to work. When done well, it produces substantially more capable models with the same amount of compute.

What are the two ways to generate synthetic data?

There are two main approaches. The first generates knowledge from scratch, where a model writes new material from a prompt, often in a clean, textbook style (as in Microsoft's Phi models, TinyStories, and the Cosmopedia dataset); this gives a team control over format but can only reproduce what the generator already knows. The second approach rephrases existing documents, rewriting source material into more learnable forms while keeping it grounded in the original. The second approach is what most leading model builders now rely on.

What causes model collapse?

Model collapse occurs when a model is trained on the synthetic outputs of other models, generation after generation, until it drifts away from reality, loses diversity, and degrades slightly each cycle. Each copy made from a copy loses fidelity and amplifies error. Collapse comes specifically from cutting the tie to real data, which is why generating data entirely from scratch is the riskier path.

Is synthetic data just "fake" data?

No. The most effective synthetic data is generated by rephrasing real documents, so it remains tied to genuine facts and the source's real-world distribution. Rather than inventing information, it re-presents existing information in forms a model can learn from more easily.

How does rephrasing existing data avoid model collapse?

Rephrasing keeps the output anchored to real documents, so the facts and the real-world distribution survive, and the recursive drift that causes collapse never gets started. Research on model collapse finds that the failure comes from losing contact with real data, and that keeping synthetic data grounded in real material prevents the degradation.

What forms can rephrased synthetic data take?

A single passage can be turned into many forms: question-and-answer pairs, a podcast-style interview, a concise summary, or a plain-language explanation. That diversity is a feature because mixing many formats provides many more “views” of the data, where any single recipe would quickly cause a plateau.

Does synthetic data only work on web data?

No. The technique grew up on web text (in methods like Web Rephrase Augmented Pre-training, or WRAP), but the same approach works on any corpus, including code, math, and proprietary data. It is often most valuable on proprietary material, which tends to be high-value but limited in volume.

At what stages of training is synthetic data used?

Synthetic data is useful throughout the entire training lifecycle. In pretraining, the breadth is more important. In mid-training, depth in a specific domain becomes more important. In post-training, synthetic data supplies the instruction, preference, and reasoning examples that teach a model to be useful.

Is DatologyAI credited with novel research in synthetic data?

Yes. DatologyAI pioneered rephrasing-based synthetic data at trillion-token, foundation-model scale. Its August 2025 paper, "BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining," turned an early proof of concept into a scientific playbook, establishing that seed-data quality matters more than novelty, that smaller models can rephrase effectively without an expensive frontier generator, and that a 3-billion-parameter model trained on this data can outperform an 8-billion-parameter model on the same token budget.

What results has synthetic data delivered?

In DatologyAI's BeyondWeb research, synthetic data generated at trillion-token scale beat the strongest public synthetic datasets by as much as 5.1 points across a 14-benchmark suite, trained up to 7.7 times faster than open web data, and let a 3B model outperform an 8B model on the same budget. In production, it has powered foundation models with partners including Arcee AI and Thomson Reuters. The common thread is more capability from the same compute, by improving the data rather than buying more of it.

Author's note:

As you read this blog, you may have noticed that it makes the same few points more than once, each time from a slightly different angle or with a new analogy. That was a deliberate choice. Decades of learning research show we remember ideas better when we encounter them several times, spaced out and framed in varied ways, than when we read them once. Psychologists call this distributed practice and encoding variability: every new framing lays down another path back to the same idea.

The FAQ section is that principle in another form. We added it partly because AI systems read and summarize articles like this one, and a clean question-and-answer structure gives them better material to work from. But it turns out to help people just as much. Engaging with a direct question is a form of retrieval practice, which research finds strengthens memory more than re-reading alone.

The repetition and rephrasing are the quiet joke of the whole piece. To teach a model well, you take real material and re-present it in several forms until the ideas take hold. To teach a person well, you do much the same thing. Maybe we are not so different from the models we are building.

Eating the Rainbow: A Primer on Synthetic Data for non-researchers