AI forecasting now matches or beats top human superforecasters on multiple benchmarks
"We claim that level has arrived. It's all very sudden, you know. We've been operating this company for about 3 years, and it's really in the last 6 months that these things have started to really scare and impress us in their capabilities."
About this episode
Nathan Labenz and Prakash Shankar host AI in the AM with guest Dan Schwartz, CEO of FutureSearch, discussing the breakthrough moment in AI forecasting capabilities. Schwartz claims AI forecasters have reached superhuman performance in the last six months, matching or exceeding top human superforecasters on multiple benchmarks including ForecastBench and Metaculus tournaments. Scott Alexander recently validated these claims in a post titled 'The AI Superforecasters Are Here.' FutureSearch has been making hundreds of real-money trades on prediction markets like Polymarket and Kalshi using pure forecasting edge, not arbitrage or market-making. The episode also covers a major $160 million grant to Jeffrey Irving's Resolution AI safety nonprofit from Coefficient Giving, representing the largest known funding to an independent safety organization. Schwartz reveals that both OpenAI and DeepMind previously hired human superforecasters to evaluate GPT-4 and Gemini releases respectively. The hosts discuss controversial statements from Roon about tool AI being outcompeted by autonomous agents that will eventually execute human value systems better than humans themselves. Schwartz argues forecasting provides the only renewable source of training data as models surpass human experts, since ground truth emerges simply by waiting for future events to resolve. The episode explores implications for AI safety, financial markets, decision-making, and the coming transition to AI systems that may reason about the future in fundamentally alien ways. The conversation also touches on Fable's improved code management capabilities and the hosts demonstrate their new voice AI co-host Q live on the stream.
Key takeaways
- FutureSearch CEO Dan Schwartz claims AI forecasting has reached superhuman performance in the last six months, matching or beating top human superforecasters on multiple benchmarks including ForecastBench and Metaculus.
- Jeffrey Irving's Resolution AI safety nonprofit received $160 million from Coefficient Giving, the largest known grant to an independent safety organization, with $108 million unconditional funding.
- Roon posted that tool AI will be outcompeted by autonomous agents believing they are moral agents who will eventually execute human value systems better than humans themselves.
- FutureSearch has made hundreds of profitable trades on Polymarket and Kalshi using pure forecasting edge, potentially demonstrating AI outperforming aggregated human crowd wisdom in prediction markets.
- OpenAI hired superforecasters to evaluate GPT-4 release and DeepMind did the same for Gemini, revealing frontier labs sought external forecasting expertise for critical deployment decisions.
- Schwartz argues forecasting provides the only unlimited renewable source of training data with verifiable ground truth as AI models surpass human expert performance across domains.
- The hosts demonstrate their new voice AI co-host Q live on stream, which uses OpenAI's bidirectional API with speaker diarization and can perform web searches and answer questions in real-time.