Eric Bigelow says RL may erode chain-of-thought readability and faithfulness

What was said

"when you're doing RL whether it's with like a single model or with sub aents there isn't necessarily an incentive to keep things human readable right like the objective is to find the to have the final if you're doing like outcome level verification instead of process level verification then your objective is to basically have a better final answer by whatever means possible."

Eric Bigelow argues that reinforcement learning can drive chain-of-thought text and agent-to-agent messages towards strange dialects because it rewards strong final answers, not human-readable intermediate steps. He also says that optimising chain-of-thought for monitorability could reduce faithfulness if models learn to produce plausible-looking steps that meet the objective.