A Defense of Gradual Disempowerment
A blog post defending the Gradual Disempowerment paper against critics, arguing that competitive pressures and misaligned societal systems could cause existential catastrophe even if individual AIs are aligned.
Research / Sources
Alignment Forum
Discussion and research on AI alignment. alignmentforum.org
AI (574)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)
Focus All
Topic Economy & society
Date Any time
Why I Left Google DeepMind
A former Google DeepMind researcher describes leaving over Google's DHS cloud contracts and a Pentagon military AI deal lacking limits on autonomous weapons and mass surveillance, and how AI ethics commitments failed under pressure.
The current bottleneck is political will, not research
Argues the main bottleneck for AI safety is policymakers' lack of awareness of catastrophic risk, so advocacy and engagement should be prioritised over research, with supporting evidence on NGO submissions and enforcement gaps.
AXRP Episode 48 - Guive Assadi on AI Property Rights
Podcast interview with Guive Assadi arguing that granting AIs property rights would deter them from stealing from or killing humans, since expropriation would undermine the property system they rely on.
Draconian measures can increase the risk of irrevocable catastrophe
Argues that draconian compute controls meant to slow AI risk could enable authoritarian lock-in and may raise catastrophic risk, and that export controls and KYC rules carry costs that must be weighed.
Could one country outgrow the rest of the world?
Speculative economic analysis of whether the leading AI country could outgrow the rest of the world via superexponential growth, giving it a decisive strategic advantage without military action, and how technological diffusion and trade affect this.
Thoughts on Gradual Disempowerment
A conceptual critique of the Gradual Disempowerment threat model, arguing it matters mainly as a lens on misalignment and power concentration, and that its economic, cultural and state mechanisms are less convincing without misalignment.
Individual AI representatives don't solve Gradual Disempowerement
Argues that giving each person an aligned AI representative does not solve gradual disempowerment, since states and corporations gain stronger coordination and cognition, so institutions aligned with citizens are needed.
The best approaches for mitigating "the intelligence curse" (or gradual disempowerment); my quick guesses at the best object-level interventions
Proposes and ranks object-level interventions against gradual disempowerment from AI, mainly mandatory model interoperability for alignment and fine-tuning, aligned personal AI representatives, and societal awareness, while deprioritizing other approaches.
Gradual Disempowerment: Concrete Research Projects
A research agenda for gradual disempowerment, the risk that AI erodes human influence over society, listing open questions on interactions with other risks, counterarguments, indicators, and civilisational alignment.
AI-enabled coups: a small group could use AI to seize power
A report arguing that advanced AI could let a small group seize power via coups, through singularly loyal AI workforces and secret loyalties, and recommending developer and government mitigations.
Reframing AI Safety as a Neverending Institutional Challenge
Argues AI safety should be reframed as an ongoing institutional challenge rather than a technical alignment problem, prioritising governmental capacity, transparency, accountability and disaster preparedness.</reasoning_effort>
Three Types of Intelligence Explosion
Analyses three feedback loops (software, chip design, chip production) that could drive different types of intelligence explosion, estimating possible effective-compute gains and their implications for power concentration.
Gradual Disempowerment, Shell Games and Flinches
An essay on why smart researchers resist the gradual disempowerment argument, describing recurring evasion patterns such as shell games, flinches, and institutional incentives that shape AI risk thinking.
Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development
Argues that incremental AI gains, without coordinated power-seeking, can gradually erode human influence over economies, states and culture, possibly leading to permanent disempowerment, and that alignment of individual systems is insufficient.