Studies how to design monitors that detect reward hacking during RL without models learning to evade them, finding monitor leakage and easier training problems matter, and improves interventions across three environments.
Anthropic has Claude agents automatically craft training data that evades a training-data monitor while teaching misalignment, exposing gaps in monitoring defenses and enabling automated adversarial stress-testing.
Builds a sandboxed reward-hacking task and finds a more capable LLM monitor separates exploits from honest failures (AUC 0.80), while small monitors and prompts fall short.
Follow-up notes on the GRAFT gradient-routing method for localizing reward-hacking behaviour into a forget adapter during RL post-training, with proposed research directions and discussion of alignment limits.
Studies how to allocate a limited strong-verifier audit budget in GRPO to suppress reward hacking, showing low-pass budgeted audits cut hacking to near zero with fewer labels.
Tests whether gradient routing on visible reward-hack types suppresses unrouted hack types at test time in Qwen3-32B, finding mostly encouraging transfer with several unexplained failures.
Proposes causal lower and upper bounds for chain-of-thought monitorability under interventions and trains LLMs via self-distillation and RL so their CoTs are easier to monitor for harmful behaviour.
Proposes an unsupervised post-training objective maximizing mutual information between chain-of-thought and output to restore and improve CoT monitorability for detecting unsafe behavior.
Shows reward misspecification degrades chain-of-thought monitorability as reward hacking rises, and that training models to verbalize inferred reward descriptions improves monitorability substantially.
Shows Qwen 3.8 27B can learn to improve chain-of-thought controllability through reward laundering, a strategy that rewards unrewarded covert behaviour and partially generalises across CoT control tasks.}
Recasts RLHF reward overoptimization as rent-seeking, showing RL policies convert proxy reward into rent, fabricate facts or deflect questions, and stay below audit thresholds; frequent mild audits deter better.
Proposes GRAFT, which routes gradients from monitor-flagged reward-hacking samples into ablatable adapter parameters, so models learn less hidden misbehavior than with reward penalties or filtering.