AI:AM #4: Cameron on Model Consciousness, Duvenaud's Gradual Disempowerment, swyx's AI-Eng Alpha

In this weekly highlights compilation from AI in the AM, hosts Prakash and Nathan explore the frontier of AI through conversations with researchers and industry leaders grappling with questions of consciousness, control, and consequence. The episode opens with Cameron Berg of Reciprocal Research presenting startling findings that Frontier LLMs score 30% on consciousness-relevant features when evaluated against leading neuroscience theories, approaching the 46% score of bees. Berg's methodology uses AI judges to assess whether model architectures contain computational substrates predicted by theories like global workspace theory, with scores rising to 40-45% when models operate as agents. The conversation then shifts to civilizational scale with David Dubinot from University of Toronto, who argues that even perfectly aligned AI leads to gradual human disempowerment. He assigns 80% probability to a scenario where humans lose all economic comparative advantage and face resource starvation, comparing humanity's position to monkeys watching humans build cities while assuming the banana economy will remain central. Google DeepMind's Michiel Bakker warns that Europe faces total AI dependence on the US by 2031, caught between regulating away access and sitting under an umbrella that may not protect economic interests. The technical deep dives include Bing Xu revealing that NVIDIA's CUDA moat actually strengthens in the agent era because superior tooling enables faster AI-driven optimization, with his company using GPT-4.5-powered swarms to evolve GPU kernels achieving 50% speedups on new workloads. Swyx from Latent Space discusses how Frontier Code benchmark reveals 50% of passing code is unmergeable, while Eric Olson from ConsenSys explains science applications can retain 95% of Frontier model performance using billion-parameter models through careful distillation. The episode closes with Eric Vaughn of IgniteTech reporting 80% employee turnover after forcing company-wide AI adoption, yet achieving profitability on a major acquisition through AI-native operations including automated interviewing and email personas.

• Berg's research using AI judges found Frontier LLMs score 30% on consciousness-relevant computational features, rising to 40-45% as agents, approaching bee-level scores of 46%.
• Dubinot assigns 80% probability to gradual human disempowerment scenario where AI eliminates all comparative advantage, potentially leading to starvation rather than mere unemployment.
• Bakker warns Europe cannot regulate AI without domestic champions and risks being cut off from Frontier models if compliance costs exceed fractional European revenue value.
• Xu claims NVIDIA's CUDA moat deepens in agent era because superior profiling tools enable GPT-4.5-powered kernel optimization swarms to achieve 50% speedups on new workloads.
• Vaughn reports 80% employee turnover at IgniteTech after mandating AI adoption, yet achieved profitability on nine-digit acquisition using AI for interviewing, communication and development.
• Swyx reveals Cognition's Frontier Code benchmark shows 50% of code passing standard tests is unmergeable, driving focus on maintainability rubrics over raw test passage.
• Olson demonstrates specialized billion-parameter models can retain 95% of Frontier performance through distillation for narrow classification tasks with sub-0.1 second latency.