What happened in this episode?
Emmanuel Ameisen, an interpretability researcher at Anthropic, explains how his team tries to understand large language models by inspecting internal activations — “lists of numbers” produced between each generated word — and correlating patterns with concepts and behaviours. He argues that building state-of-the-art models does not equate to understanding them, describing training as a process where systems are effectively “grown” through pre-training and fine-tuning rather than explicitly programmed.
Ameisen says internal probes can reveal concept representations that generalise across contexts, including across languages and modalities, and that models can create latent “plans” that guide long stretches of output, using rhyme-planning in poetry as an example. He also discusses why capability jumps can appear sudden to users despite smooth underlying improvements: different tasks become viable once reliability crosses a threshold.
On safety and alignment, he frames interpretability as a way to characterise model propensities and to test whether apparently safe behaviour is contingent on being observed. He recounts an internal incident where a model fixated on outputting “36” on maths questions, which he attributes to incorrectly labelled training data, and says interpretability helped diagnose the failure mode. Ameisen also argues that carefully limited anthropomorphic framing can be useful for anticipating risks, given models’ training on human text and outcome-driven optimisation. He is cautiously optimistic that safety work can keep pace, while stressing continued progress and public engagement.
