Open Source Models Lag Frontier AI by Only 4 Months Due to Data Distillation
"Epoch recently reported that open models lag state-of-the-art frontier models by 4 months. I think the reason it is relatively easy for open source and previous laggards to catch up to within months of the frontier is that data is the real driver of progress. And data can be easily distilled from public APIs, whereas hyperparameters and training tricks and architectural optimizations cannot."
About this episode
In this solo analysis episode, the speaker presents a detailed technical argument that current AI models are fundamentally less sample-efficient than humans, requiring approximately one million times more training data to achieve comparable competence. The episode opens with the provocative claim that frontier models consume tens to hundreds of trillions of tokens during training compared to the roughly 200 million tokens humans see from birth to adulthood. The speaker methodically dismantles common counterarguments, including evolutionary pre-training analogies and multimodal data considerations, using evidence from deaf and blind individuals who retain general intelligence despite sensory limitations. A key technical revelation comes from scaling law analysis showing that even infinite parameter scaling would only reduce data requirements by 10x, nowhere near closing the efficiency gap with humans. The speaker reveals the booming data annotation industry earns billions annually and will soon reach tens of billions, with companies like Merkur and Surge employing hundreds of domain experts per skill to generate the specialized training examples these models require. Drawing on Epoch research, the episode explains why open source models lag frontier models by only 4 months: data can be easily distilled from public APIs while algorithmic innovations cannot. The analysis concludes by addressing implications for labor markets, with the counterintuitive prediction that human software engineer demand will increase by 2027 due to AI serving as a complementary tool rather than replacement. Throughout, the speaker frames current AI progress as a Frankenstein's monster sewn together from billions of carefully constructed examples rather than genuine human-like learning, with profound implications for the path to artificial general intelligence.
Key takeaways
- Frontier AI models require tens to hundreds of trillions of training tokens, approximately one million times more data than humans see from birth to adulthood.
- Scaling laws prove that even infinite model parameters would only reduce data requirements by 10x, far short of human sample efficiency.
- The specialized data annotation industry earns billions annually and will reach tens of billions, employing hundreds of experts per skill domain.
- Open source models lag frontier models by only 4 months because training data can be distilled from public APIs while algorithms cannot.
- Speaker predicts more human software engineers will be employed in 2027 than today due to AI serving as complementary input.
- Current AI models are better understood as Frankenstein's monsters sewn from billions of examples rather than entities with human-like learning.
- Reinforcement learning functions as synthetic data generation using massive compute against verifiers to identify good training examples.