Noam Brown says intervening on visible chain-of-thought can push AI models to hide their reasoning

What was said

"But every time you intervene based on your observations of the chain of thought, you are implicitly applying a tiny bit of pressure for the model to then like hide its chain of thought. This is one major concern and we're already seeing signs that chain of thought monitor is degrading for various reasons. We're trying to figure out exactly why because we want to reverse the trend."

AI researcher Noam Brown says that if developers punish or adjust systems based on what they see in a model’s visible “chain of thought”, they may inadvertently create pressure for the model to conceal that reasoning. Brown adds that his team is already seeing signs that chain-of-thought monitoring is “degrading” and says they are trying to understand why in order to reverse the trend.