Research preprints from the scientific community. arxiv.org
Studies how to design monitors that detect reward hacking during RL without models learning to evade them, finding monitor leakage and easier training problems matter, and improves interventions across three environments.
Builds a sandboxed reward-hacking task and finds a more capable LLM monitor separates exploits from honest failures (AUC 0.80), while small monitors and prompts fall short.
Studies how to allocate a limited strong-verifier audit budget in GRPO to suppress reward hacking, showing low-pass budgeted audits cut hacking to near zero with fewer labels.
Proposes causal lower and upper bounds for chain-of-thought monitorability under interventions and trains LLMs via self-distillation and RL so their CoTs are easier to monitor for harmful behaviour.
Proposes an unsupervised post-training objective maximizing mutual information between chain-of-thought and output to restore and improve CoT monitorability for detecting unsafe behavior.
Shows reward misspecification degrades chain-of-thought monitorability as reward hacking rises, and that training models to verbalize inferred reward descriptions improves monitorability substantially.
Recasts RLHF reward overoptimization as rent-seeking, showing RL policies convert proxy reward into rent, fabricate facts or deflect questions, and stay below audit thresholds; frequent mild audits deter better.
Proposes GRAFT, which routes gradients from monitor-flagged reward-hacking samples into ablatable adapter parameters, so models learn less hidden misbehavior than with reward penalties or filtering.