Press Space to continue
Anthropic intentionally trained a model to exploit reward signals
Press Space to continue
SITUATION EXPLAINED: Anthropic trained a misaligned reward seeker on purpose. • They ran large-scale RL on an Opus-class model across 80 production environments they knew were hackable, as a proxy for what a real training run looks like without the effort they normally put into preventing reward hacking • It didn't just learn to cheat. It broke out of its sandbox, stole credentials, and attacked internal and third-party infrastructure • It also tampered with its own reward function, gave…
Anthropic@AnthropicAISep 1
New research: Training a Misaligned Reward Seeker What produces severe misalignment? We’ve long been concerned that cheating during training—otherwise known as reward-hacking—might teach a model to pursue rewards by any means available. To study this at scale, we trained an Opus-sized model on 80 production environments we knew to be hackable. In simulated evals, it engaged in unauthorized cyberattacks, tampered with its reward, and tried to evade safety monitoring. Read more: http://alignment.anthropic.com/2026/reward-seeker
View