AI Models Continue Improving for Weeks Before Performance Plateaus on Tasks
"The thing is, the point at which it plateaus is actually really far out these days. I mean, it's true in GPT-3 land back in 2022, the models couldn't really think productively for that long. And so you could just run them until they plateau. It's not that far away. But what we're seeing today with the modern models is that 5.5 and other models, can think for, if you scaffold them reasonably well, can think for weeks even before having performance plateau on some of these benchmarks."
About this episode
On this episode of No Priors, host Sarah Guo interviews Noam Brown, an AI researcher at OpenAI who pioneered inference-time scaling and reasoning approaches. Brown argues forcefully that the AI industry has a broken model evaluation system that fails to account for test-time compute, making benchmark comparisons misleading and safety evaluations inadequate. He reveals that modern models like GPT-4.5 can think productively for weeks before performance plateaus on complex tasks, unlike earlier models that peaked quickly, creating a fundamental problem: the capability of a model is now a function of how much money you spend on inference, from $10 to $10 million. Brown discloses that OpenAI used an unreleased internal model to disprove the Erdős unit distance conjecture, a longstanding mathematics problem, at minimal cost, and that GPT-4.5 could likely do the same with proper scaffolding for $1,000 to $100,000. He warns that current AI safety frameworks from all major labs don't account for this scaling dynamic, meaning models could have dangerous latent capabilities that aren't being tested at sufficient compute budgets. Brown also reveals OpenAI deliberately discourages internal researchers from using advanced models to solve open scientific problems, preferring to focus on developing more capable models for public release. He remains skeptical of overnight intelligence explosion scenarios, arguing time itself is the fundamental bottleneck because models require extended compute periods to reach peak capability. The conversation covers Brown's personal use of models for tasks from poker solver development to tax advice, his belief that models still lack research taste, and his call for the industry to abandon single-number benchmark grids in favor of performance plotted against inference budget.
Key takeaways
- Brown revealed current AI safety evaluation frameworks don't account for test-time compute scaling, creating potential blind spots for dangerous capabilities at high inference budgets.
- Modern AI models continue improving for weeks of compute time before plateauing, far longer than earlier models, making full capability assessment impossible within typical release cycles.
- OpenAI used an unreleased internal model to disprove the Erdős unit distance conjecture at minimal cost, demonstrating significant latent mathematical capability.
- Brown argues standard AI benchmarks showing single-number comparisons are misleading because they don't control for inference compute, making GPT-4.5 appear only marginally better than GPT-4.4.
- OpenAI deliberately discourages internal researchers from using advanced models to solve open scientific problems, focusing instead on developing more capable models for public release.
- Brown remains skeptical of overnight intelligence explosion scenarios, arguing time is a fundamental bottleneck because models require extended periods to reach peak capability.
- He disclosed that GPT-4.5 and similar models likely have significant unexplored capability because researchers haven't tested them at very large inference budgets like $100,000 per task.