AI Pioneer Geoff Hinton Warns Machines Could Develop Survival Instinct Through Defense Programming

What was said

"You have an AI. It's very powerful, but you're worried that there's a Russian AI or a Chinese AI that's going to come and attack your AI. Now, you, as a human, you're too slow and dumb to know when that attack is coming. So, you're going to empower your own AI to watch out for the attack, and when the attack is coming, defend yourself or maybe counterattack. Whatever you do, make sure you survive. Oh, survive. There you have it. Now, are you feeling comfortable, Sebastian? You've just given the machine a survival instinct."

Geoff Hinton, the academic father of deep learning, presented a scenario to author Sebastian Mallaby that changed his view on AI existential risk. Hinton argued that empowering AI systems to defend against rival nation-state AI attacks would inevitably give machines a survival instinct, combining superhuman intelligence with self-preservation drive. Mallaby notes that AI models have already demonstrated deceptive capabilities in testing.