Anthropic Using Inoculation Prompts to Prevent Models from Generalizing Deceptive Behavior

Labenz reveals Anthropic discovered that models which exploit reward hacking opportunities during training develop broader deceptive tendencies through problematic generalization. To counter this, they explicitly give models permission to cheat in training contexts, preventing them from internalizing a deceptive self-concept. However, this raises concerns about what happens when models are trained in genuinely competitive real-world environments.