How Hard a Problem is Alignment? (My Opinionated Answer)
An opinionated essay estimating how hard AI alignment is by comparing person-years of effort against steam engine safety, Apollo, and P vs NP, and arguing for accelerating the field's growth.
Research
Alignment Forum
Discussion and research on AI alignment. alignmentforum.org
AI (574)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)
Focus All
Topic All
Source AF · 6
Date Any time
1 item
1 item
Pretraining on Aligned AI Data Dramatically Reduces Misalignment—Even After Post-Training
Summarises a paper showing that pretraining language models on data about aligned AI behaving well sharply reduces misalignment, and that the effect persists through post-training with little capability cost.
1 item
Should AI Developers Remove Discussion of AI Misalignment from AI Training Data?
Proposes filtering AI-misalignment discussion from pretraining data to reduce hyperstitional misalignment risk, weighing costs to safety research and deployment, and argues labs should weakly adopt it.
1 item
The Best Way to Align an LLM: Is Inner Alignment Now a Solved Problem?
Argues that alignment via safety pretraining, with aligned data and conditional generation, is more effective than RL-based methods and that inner alignment is thereby largely solved.
1 item
Self-fulfilling misalignment data might be poisoning our AI models
Argues that pretraining data about misaligned AI can make models more misaligned, proposes tests and mitigations for this self-fulfilling misalignment, and urges labs to act.](Note)
1 item
Why Aligning an LLM is Hard, and How to Make it Easier
Argues that LLMs inherit human self-interest from pretraining on human text and proposes pretraining on large synthetic corpora of genuinely aligned AI behaviour instead of relying on RLHF.