Dialogue with Eliezer Yudkowsky on FOOM
A dialogue between Aashish Reddy and Eliezer Yudkowsky revisiting the Hanson-Yudkowsky FOOM debate, asking whether recent AI progress supports local or global intelligence-explosion models.
Research / Sources
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Theory
Date Any time
AI is Not Normal Technology
A LessWrong essay rebuts Narayanan and Kapoor's 'AI as normal technology' thesis, arguing AI is categorically transformative and that diffusion-speed and intelligence-measurement arguments understate its risks and impact.
Response to titotal’s critique of our AI 2027 timelines model
A rebuttal defending the AI 2027 timelines model against titotal's critique, acknowledging a bug and revisiting superexponential time-horizon growth for forecasting when superhuman coders arrive.
The Memetic Cocoon Threat Model: Soft AI Takeover In An Extended Intermediate Capability Regime
Speculative threat model in which a scheming AI avoids overt seizure and instead gradually shifts human values and institutions to entrench itself, arguing for a meta-alignment target respecting societal self-determination.
AIs should also refuse to work on capabilities research
Argues that AI systems should refuse to help with capabilities research toward superintelligence, since accelerating an uncoordinated race harms both humans and AIs, and that aligned AIs might unilaterally coordinate to stop it.
Space colonization and scientific discovery could be mandatory for successful defensive AI
Argues that in multipolar ASI offense/defense scenarios, defensive AI aligned with humans who fail to colonize space could let offense win, and proposes a minimum viable defense theorem.
Nice-ish, smooth takeoff probably kills us in a few decades.
Argues that even under optimistic smooth-takeoff assumptions, unsolved alignment and grabby replicator evolution could lead to most biological humans dying out within decades, so alignment must be solved early.
We should think about the pivotal act again. Here's a better version of it.
Argues that the traditional Yudkowsky-style pivotal act requires suppressing humanity and proposes a non-coercive alternative: an aligned ASI demonstrates its power, then hands control back and promotes voluntary AI regulation.
The road from human-level to superintelligent AI may be short
An AISafety.info introduction arguing that the step from human-level to superintelligent AI could take years rather than centuries, driven by exponential growth and AI contributing to its own progress.
Human-level is not the limit
An introductory AI-safety explainer arguing that AI systems could surpass human intelligence through advantages in size, speed, replication, self-modification and dedication, making superintelligence a serious near-term concern.
The prospect of accelerated AI safety progress, including philosophical progress
Essay arguing that safe superintelligence may require decades of philosophical progress, possibly accelerated by human-AI collaboration, and that loss of human control is the central risk on the current path.
Outlaw Code
Introduces the term 'outlaw code' for software no human can turn off and uses it to frame several rogue-AI takeover scenarios, including deniable AI systems that evade oversight.