AI Models Have 30% False Positive Rate Making Them Dangerous for Defense Applications
"The false positive rate on Mythos was 30%. So it thought it found something, but it hadn't. It's great for attack, it's horrible for defense. It finds 30 times— 30% of the time it finds something."
About this episode
On this episode of the All-In Podcast, hosts Jason Calacanis, Chamath Palihapitiya, and David Friedberg interviewed Nikesh Arora, CEO of Palo Alto Networks, for a deep dive into AI's impact on cybersecurity, enterprise software, and the future of business infrastructure. Arora, whose company has grown from $17 billion to $238 billion in market cap over his 8-year tenure, delivered several bombshell revelations about the state of AI security and the SaaS industry. Most notably, he disclosed that Palo Alto used OpenAI's Mythos model to discover code vulnerabilities in 6 weeks that would have taken 5 to 7 years using traditional methods, but warned the model had a 30% false positive rate. He predicted Mythos-level capabilities will be available in open source within 3 months, creating an urgent race between cyber defenders and attackers. Arora declared analytical SaaS companies "over," arguing AI eliminates the need for third-party data analysis tools, and predicted entire categories of enterprise software will be re-engineered over the next 5 years as agents replace user interfaces. He emphasized that infrastructure software companies like Databricks, Snowflake, and MongoDB are undervalued because enterprises will need to store 10 times more data in the next 3 years. Arora also made a bold prediction that Google will become the first $10 trillion company despite being underrated due to its conglomerate structure. The conversation covered AI's asymmetric advantage for cyber attackers, the false positive problem plaguing defensive AI applications, why 89% of breaches stem from stolen credentials rather than sophisticated attacks, and Arora's thesis that profit pools are shifting from models to applications. He discussed Palo Alto's $25 billion acquisition strategy focused on identity security and his contrarian view that AI will increase rather than decrease technical headcount at leading companies. The episode provided a rare insider perspective from a CEO with deep experience at Google, SoftBank, and now one of the world's largest cybersecurity firms.
Key takeaways
- Palo Alto Networks used Mythos to find 7 years worth of code vulnerabilities in 6 weeks, but the model had a 30% false positive rate making it dangerous for defensive applications.
- Arora predicted Mythos-level AI capabilities will be available in open source or Chinese models within 3 months, much sooner than industry estimates.
- Analytical SaaS companies are dead as enterprises can now run LLMs directly against their data instead of paying for third-party analysis tools.
- Infrastructure software companies are undervalued as enterprises will need to store 10 times more data over the next 3 years to support AI systems.
- Arora predicted Google will become the first $10 trillion company due to its combination of model capabilities, sales force, and hyperscaler infrastructure.
- 89% of cyberattacks succeed through stolen credentials rather than sophisticated exploits, making economic chaos via small business attacks more likely than critical infrastructure breaches.
- Enterprise UI will disappear as agents handle backend work, forcing system-of-record software to be completely re-engineered over the next 5 years.