The Magnus Challenge
An analogy-based essay arguing that aligning AI through a chain of successively stronger aligned models resembles the chess proxy problem, with a pause as a timing lever, and that the chain must be very reliable.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Strategy & forecasting + Oversight
Source LessWrong · 7
Date Any time
1 item
1 item
Distillation of the AI2040 Alignment Roadmap
A distillation of the AI2040 alignment roadmap, laying out a Plan C strategy for a responsible frontier lab: phases of mitigation, pause, handoff to AIs at a Min-H capability threshold, and alternative exit plans.
1 item
Where does the race to automate AI research end?
Argues that automating AI research could cause an unrecoverable alignment failure, because oversight breaks down at scale, capabilities self-amplify, and capabilities outpace alignment.
1 item
Agenda Reflection: Testing Automated Alignment
Proposes an empirical agenda testing whether AI safety researchers can detect flawed automated-alignment outputs, and argues early successes may breed overconfidence and call for governance lines near handover to AI.
1 item
AI Safety Isn't So Unique
Argues AI safety research is not uniquely hard to automate, since verifying research value and needing conceptual breakthroughs are common across fields, so labs have incentives to solve automated-researcher training.
1 item
Why would AI companies use human-level AI to do alignment research?
Argues that AI companies are unlikely to actually devote human-level AI systems to alignment research under alignment bootstrapping plans, since they would prioritize capabilities as they do with human researchers.
1 item
Beliefs and state of mind into 2025
The author sets out strategic beliefs on AI safety: when to pause, why alignment research shows diminishing returns, and using a 10x AI-researcher system to help solve alignment.