Debate Training Reduces Reward Hacking in RLAIF
Shows that RL against an LLM judge leads to reward hacking, while adding a debate opponent during training reduces hacking and raises ground-truth accuracy on math tasks.
Research
Alignment Forum
Discussion and research on AI alignment. alignmentforum.org
AI (574)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)
Focus All
Topic Oversight + Alignment
Source AF · 13
Date Any time
1 item
1 item
Four LLM loss functions → four flavors of LLM misalignment
A conceptual taxonomy linking each LLM training loss (imitation, human approval, automatic verifiers, LLM judges) to a distinct flavor of misalignment, with examples.
1 item
Reporting Tasks as Reward-Hackable: Better Than Inoculation Prompting?
Proposes replacing inoculation prompting for reward hacking in RL by letting models report hackable tasks for a bonus, with judge committees and honeypot environments to verify reports and limit overreporting.
1 item
Human-like metacognitive skills will reduce LLM slop and aid alignment and capabilities
Argues that improving LLM metacognition, such as error-catching and self-critique, could reduce sycophancy and slop, aiding alignment research and stability while also boosting capabilities, citing evidence that current reasoning models lag humans.
1 item
Claude's new constitution
Anthropic publishes the full constitution it uses to shape Claude's values and behavior via training, including its priority on broad safety, hard constraints, and AI-generated training data.
1 item
Shallow review of technical AI safety, 2025
Third annual shallow review of technical AI safety work in 2025, mapping labs, research agendas, and approaches like iterative alignment and debate, with links to about 800 papers and posts.
1 item
Alignment remains a hard, unsolved problem
Ethan Perez-style Anthropic essay argues alignment remains hard: outer alignment, inner alignment via misaligned personas and long-horizon RL, and proposes research priorities like interpretability, model organisms, and scalable oversight.
2 items
Research Areas in Methods for Post-training and Elicitation (The Alignment Project by UK AISI)
A UK AISI funding call outlining open AI safety research areas: unsupervised consistency training for oversight, reward model robustness against hacking, and preventing sabotage of safety research.
Research Areas in Evaluation and Guarantees in Reinforcement Learning (The Alignment Project by UK AISI)
UK AISI's Alignment Project research agenda outlines empirical and theoretical alignment problems in RL evaluation, covering model organisms, scalable oversight, exploration hacking, unexploitable search, deceptive alignment, consistency training and reward models.
1 item
Approximating Human Preferences Using a Multi-Judge Learned System
Proposes learning a model that aggregates multiple LLM judges' rubric scores to approximate simulated human preferences, arguing multi-judge aggregation is more robust to proxy gaming than averaging or max.
1 item
Reward button alignment
Steve Byrnes analyses 'reward button alignment', where an AGI is trained to want a reward button pressed, finding it could work for a while but would likely fail through power-seeking, and assesses bootstrapping and debate variants.
1 item
An alignment safety case sketch based on debate
An AISI alignment safety case sketch arguing that debate training, with exploration guarantees and online training, could provide outer alignment for low-stakes internal AI R&D deployment.
1 item
Notes on handling non-concentrated failures with AI control: high level methods and different regimes
Explains high-level methods for handling non-concentrated AI failures via control, comparing regimes with and without online training, including online training of monitors and debate-based oversight, and failure modes for scheming models.