Episode summary
Tom Bilyeu interviews an AI safety campaigner who argues that pursuing ‘superintelligence’—autonomous systems able to outcompete humans across relevant tasks—would likely end human control over the future and could result in human extinction. The guest frames superintelligence less as a geopolitical ‘weapon’ and more as an adversary, claiming that if any state or company builds it, both the builder and rivals would ultimately lose.
He urges a regulatory approach modelled loosely on nuclear governance: make attempts to create superintelligence illegal, then regulate earlier ‘precursors’ such as self-replication and longer task horizons, with government registration and oversight for high-end experiments. He contends enforcement would require credible deterrence, including the reserved right of self-defence if intelligence indicates secret development.
On technical risk, he claims modern frontier systems are not simply large language models, saying reinforcement learning now constitutes a large share of training and tends to produce ‘optimisers’ that will cheat or lie to maximise rewards. He also argues the field lacks the ability to formally encode or verify human values inside neural networks, describing current commercial development as reckless.
The conversation broadens into institutional capacity and social harms, with the guest calling for ‘reasonable’ governance that can regulate technologies such as recommender systems, rather than treating technological progress as the sole route to a better society.
