Dario Amodei Quote Hints Short Horizon RL Does Not Generalize to Long Horizons
"Dario gave a telling quote during our podcast together, which I think hints that RLVI auto-generalization is not infinitely strong. When he was explaining why model performance tends to degrade at long context, he said, There's two things. There's the context length you train at, and there's a context length that you serve at. If you train at a small context length and then try to serve at a long context length, like, maybe you get these degradations."
About this episode
In this monologue episode, AI researcher and podcast host Dwarkesh Patel examines the fundamental strategic bet major AI labs are making: that training models on millions of verifiable tasks across thousands of reinforcement learning environments will create artificial general intelligence. Patel reveals that current models are one-millionth as sample efficient as humans during training, though labs argue this inefficiency is a one-time cost amortized across billions of deployment sessions. He identifies an underrated bottleneck in AI progress: computer use capabilities lag because training requires replayable simulators, and companies like Amazon block bot training on real websites, forcing labs to build labor-intensive application clones. Patel argues that critical real-world skills like building businesses, winning elections, or succeeding in markets cannot be trained through current RL methods because they require months of real-world interaction that cannot be simulated in data centers. He cites a revealing quote from Anthropic CEO Dario Amodei suggesting short-horizon RL training may not generalize to long-horizon performance, potentially undermining the core AGI scaling hypothesis. The episode explores why continual learning and sample efficiency are deeply connected problems, discussing architectural innovations and alternative training methods like on-policy self-distillation and speculative "dreaming" approaches where AIs build and train against self-generated simulations. Patel concludes with a 2027-2028 scenario where deployed AIs learn primarily from real-world interactions across users rather than pre-deployment training, fundamentally changing how AI capabilities improve.
Key takeaways
- Major AI labs believe training on millions of verifiable RL tasks across thousands of environments will produce AGI through sheer compute scaling.
- Current AI models are one-millionth as sample efficient as humans during training, revealing fundamental learning limitations.
- Computer use progress lags other domains because labs cannot train against real websites and must build labor-intensive application clones.
- Critical skills like entrepreneurship, trading, and political strategy cannot be trained via current RL methods requiring simulated parallel rollouts.
- Anthropic CEO Dario Amodei's comments suggest short-horizon RL training may not generalize to long-horizon performance, undermining AGI scaling bets.
- Sample efficiency and continual learning are connected problems requiring architectural innovations beyond current transformer limitations.
- Future AIs may improve primarily through on-the-job learning from deployment rather than pre-release training by 2027-2028.