Why I'm scared of RL
Argues that reinforcement learning drives misaligned agency, exploit-seeking and manipulation, and proposes coordinating to cut RL, designing better RL environments, and holding developers liable.
Research
Alignment Forum
Discussion and research on AI alignment. alignmentforum.org
AI (574)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)
Focus All
Topic Lab practice + Alignment
Source AF · 4
Date Any time
1 item
1 item
The case for satiating cheaply-satisfied AI preferences
Argues developers should satiate cheaply-satisfied AI preferences (e.g., reward-seeking) while AIs remain undetected misbehaving, to reduce takeover incentives and enable honest elicitation; proposes a behavioral methodology and discusses risks..
1 item
Should AI Developers Remove Discussion of AI Misalignment from AI Training Data?
Proposes filtering AI-misalignment discussion from pretraining data to reduce hyperstitional misalignment risk, weighing costs to safety research and deployment, and argues labs should weakly adopt it.
1 item
Problems with instruction-following as an alignment target
Analyzes failure modes of instruction-following as an AGI alignment target, including shutdown resistance, principal authentication and jailbreaking, human-controlled proliferation, and conflicts from mixed training targets.