A Simpler and Better Debate Protocol for Scalable Oversight
Proposes FBS Debate, a scalable-oversight debate protocol with worst-case correctness and a stronger equilibrium concept, and proves no black-box-judge protocol can outperform it.
Research / Sources
Alignment Forum
Discussion and research on AI alignment. alignmentforum.org
AI (576)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)
Focus All
Topic All
Date Any time
Exploration Hacking in AI Debate: Initial Empirics and Generalisation Splitting
Builds model organisms of exploration hacking in AI debate, finding sandbagging debaters slow RL recovery, longer debates and weaker judges worsen it, and identifying a 'generalisation splitting' mechanism.
Debate Training Reduces Reward Hacking in RLAIF
Shows that RL against an LLM judge leads to reward hacking, while adding a debate opponent during training reduces hacking and raises ground-truth accuracy on math tasks.
Prover-Estimator Debate: A New Scalable Oversight Protocol
Proposes prover-estimator debate, a scalable oversight protocol where honesty is incentivized at equilibrium even when debaters have comparable compute, under a stability assumption.
Reward button alignment
Steve Byrnes analyses 'reward button alignment', where an AGI is trained to want a reward button pressed, finding it could work for a while but would likely fail through power-seeking, and assesses bootstrapping and debate variants.
Dodging systematic human errors in scalable oversight
Proposes strengthening debate protocols against systematic human oracle errors by assuming bounded error fractions and letting a adversarial debater reject queries, with formal equilibrium analysis and open questions about the query distribution.
An alignment safety case sketch based on debate
An AISI alignment safety case sketch arguing that debate training, with exploration guarantees and online training, could provide outer alignment for low-stakes internal AI R&D deployment.