Goodfire’s Eric Bigelow says a punctuation token can sometimes change an LLM’s eventual answer

What was said

"there's one example we have in the paper where it's like kilowatt hours open in parenthesis kWh and you know you never expect that to really change how the final answer plays out. But if you actually go to that exact token with this model and you just resample a bunch of text generations, you see that in cases where it's not an open parenthesis, where it starts to be some other just arbitrary word, it ends up being a different final answer."

Eric Bigelow, a Goodfire researcher, describes a finding from his paper in which resampling at a specific point in a model’s generated reasoning showed that an apparently arbitrary token—such as an opening parenthesis in “kilowatt hours (kWh)”—can coincide with later rollouts producing a different final answer.