Jankily controlling superintelligence
Argues AI control can modestly reduce risk from misaligned superhuman AI but is likely to become hopeless within a few years of AI R&D automation, estimating how much time control can buy.
Research
Alignment Forum
Discussion and research on AI alignment. alignmentforum.org
AI (574)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)
Focus All
Topic Strategy & forecasting + Control
Source AF · 8
Date Any time
1 item
1 item
Problems with instruction-following as an alignment target
Analyzes failure modes of instruction-following as an AGI alignment target, including shutdown resistance, principal authentication and jailbreaking, human-controlled proliferation, and conflicts from mixed training targets.
1 item
Prioritizing threats for AI control
Prioritizes threats for AI control, arguing that undetected rogue internal deployments with escalated permissions are worse than self-exfiltration and should be the top priority to prevent or catch.
1 item
AI Control May Increase Existential Risk
Argues that AI control may raise existential risk by shifting outcomes toward ineffective warning shots and takeover, since controlled incidents are unlikely to prompt serious action on alignment.
1 item
On the Rationality of Deterring ASI
Presents the Superintelligence Strategy framework proposing Mutual Assured AI Malfunction deterrence, nonproliferation via compute security, and national competitiveness measures to manage catastrophic superintelligence risks.
1 item
Ten people on the inside
Argues that a small internal safety team in a rushed, unreasonable AI lab should prioritise cheap, low-overhead control measures and external research support, and that such pessimistic scenarios deserve more planning focus.
1 item
AI companies are unlikely to make high-assurance safety cases if timelines are short
Argues frontier AI companies are unlikely to produce high-assurance safety cases for existentially dangerous AI within short timelines, due to security, scheming-detection, and acceleration difficulties, and unlikely to pause unilaterally.
1 item
What’s the short timeline plan?
Apollo Research staff argue AI labs should publish detailed short-timeline (2027) plans, proposing minimal requirements: secure model weights and monitoring of scheming AIs, with faithful chain-of-thought and monitoring as core defenses.