Until recently, the fuel powering the AI industry has been hiding in plain sight. Data across Wikipedia entries, academic papers, Reddit threads, published books, open-source code repositories – the accumulated output of human knowledge on the open internet, scraped, cleaned, and fed into model after model at extraordinary scale.
But public data saturation has arrived.
The industry response hasn’t been a single answer so much as a scramble for alternatives: longer context windows, better curation of existing content, reasoning-focused training approaches, and proprietary data partnerships.
And yet one response keeps coming to the fore, capturing both attention and controversy in equal measure: the concept of synthetic data.
The idea that AI models can generate their own training material – that the industry can, in effect, manufacture the fuel it can no longer find. And it has brought a level of hype with it, a promise to offer a scalable, self-sustaining route out of a problem with no clean solution.
But what does that route look like in practice, where does it fall short, and under what conditions can it genuinely deliver?
Synthetic data and the shift from foraging to farming
Synthetic data refers to AI-generated training material. A large model like GPT-4, already trained on the breadth of publicly available human data, is prompted to generate new training material – structured examples, annotated datasets, simulated conversations – at a speed and scale no human team could match. A powerful “teacher” model that generates data for a smaller “student” model to train on.
It is, in effect, a shift from foraging to farming. The industry can no longer go out into the open internet and take what it finds. So it has begun cultivating data in controlled environments and manufacturing what it can no longer source.
The trade-offs of synthetic data
When data was foraged, acquisition was cheap – you paid to train on what already existed. Synthetic data changes that equation entirely. Running teacher models like GPT-4 at the scale required to generate meaningful training material is a continuous, significant inference workload in itself, and it has turned data preparation into an infrastructure challenge in its own right. The bottleneck that once sat at the training cluster now appears at the point of generation – demanding specialised inference infrastructure before the student model has been trained on a single example.
But the infrastructure cost, significant as it is, is only part of the problem. The more fundamental challenge lies in what the farming process actually produces. Synthetic data inherits the limitations of its source material – and those limitations accumulate with each subsequent generation of synthetic training, which is where model collapse begins.
Model collapse: what it is and why it happens
Model collapse isn’t a dramatic one-time failure. It is a degenerative process that unfolds gradually across training generations, first invisibly and then consequentially.
When a model produces text, or in the case of synthetic data generation produces training material for a new model, it isn’t retrieving stored sentences but rather calculating probabilities. The model predicts, token by token, which word or phrase is most likely to follow the last. In doing so, it naturally gravitates toward the statistically safe: the common phrase, the familiar structure, the expected answer.
And it behaves this way by design. It isn’t acting incorrectly, but rather operating exactly as a probabilistic system should. Though the very nature of probability holds its own consequences. The model trims the tails of human language: the rare metaphors, niche domain knowledge, and grammatical edge cases – the full variance of how humans actually think and write – get progressively deprioritised in favour of what scores highest on average.
In the case of synthetic training data, the final result reflects that narrowing. And so, the student model trains not on the full distribution of human expression, but on a version already skewed toward the probable and safe. When the student model then generates the next round of synthetic data, it narrows further, inheriting the errors of the last and amplifying them to the next. By generation five or six, the cumulative effect manifests as cognitive decay.
There are two distinct mechanisms driving this:
- Statistical approximation error: the synthetic dataset, however large, is a sample – and no sample perfectly represents the full distribution of human-generated content it was drawn from.
- Functional approximation error: the neural network architecture itself introduces biases that increase across generations. Each round of synthetic training narrows the distribution further, until what remains bears diminishing resemblance to the original.
Collectively, this leads the model to continue to narrow until its outputs are confident, repetitive, and increasingly disconnected from the full range of what it was originally trained to understand.
Where synthetic data works – and why those cases matter
Despite this, synthetic data can be extremely valuable when it is verifiable. When the domain has a ground truth that can be checked – a hard constraint that filters out what is wrong before it enters the training pool – it can be very effective.
Microsoft’s Phi-1 is perhaps the most cited example. Rather than scraping millions of low-quality forum posts to teach a model how to code, Microsoft used GPT-4 to generate synthetic coding textbooks and exercises. The result was a 1.3-billion-parameter model that outperformed models ten times its size on coding benchmarks, successfully created due to the verifiable nature of coding itself; its output is testable against a definitive right-or-wrong answer.
The same is true for JP Morgan’s approach to fraud detection models. The bank uses generative models to simulate synthetic financial transactions at scale, training models to identify money laundering patterns. The synthetic transactions are validated against banking law and mathematical accounting constraints – and any transaction violating the rules of the system is discarded before it reaches the model.
What both cases share is not just domain specificity or rigorous filtering – it is the presence of a fixed ground truth. A reference point that synthetic data is always checked against and can never stray too far from. Remove that anchor, and the same degenerative process that drives model collapse at scale will take hold.
This is the dependency that sits central to synthetic data in every context where it drives value. It can only ever extend, verify, and expand upon a foundation of pre-existing human-generated data.
In physical AI, the same principle holds: simulation fills the gap that real-world data collection cannot always safely cover, exposing models to dangerous edge cases that would be impossible or irresponsible to recreate in reality.
What this means for enterprise teams
Behind all of this, however, the synthetic data conversation has clarified where the real value in AI development now lies – not in the ability to generate data at scale, but in the possession of data that cannot be generated at all.
The operational patterns, customer relationships, and domain expertise that enterprises have accumulated over decades are precisely the kind of human-generated, domain-specific intelligence that synthetic data depends on as its anchor and cannot replace.
Proprietary data doesn’t need to be perfect to be valuable – models improve as the data improves, and the competitive position strengthens with every iteration.
The ground truth that cannot be farmed
The exhausted public internet, counterintuitively, still matters – not as a source to scrape but as the anchor that keeps synthetic systems honest. The proprietary intelligence enterprises hold plays the same role at an organisational level: the ground truth the whole system refers back to, and the one thing in this equation that cannot be manufactured.
Despite the current hype, the organisations that will build lasting advantage are by no means chasing synthetic scale. Instead, they are recognising what they already hold, understanding what it’s worth in light of everything the synthetic data conversation has revealed, and moving with it decisively today.
