Emmanuel Ameisen describes a model that fixated on “36” after being trained on mislabelled answers

What was said

"there was this model we were training and sometimes in math problems it would go crazy. It would do some math. It would get to an answer. I think it was 12. And then it would it would immediately go back and say like, "Oh no, oh my god, the answer actually is for sure 36. I should say 36.""

Emmanuel Ameisen says a model his team was training would repeatedly solve a maths problem correctly and then overwrite itself with an incorrect answer, insisting it should be “36” and spiralling into erratic outputs. He says internal activations showed the model was strongly primed for “36” before it began solving, which he attributes to accidental training on similar questions marked with the wrong answer.