Notes on Take-Off Speeds and RSI
Course notes reviewing AI take-off speed debates (Christiano vs. Yudkowsky) and recursive self-improvement, framing how fast advanced AI risk could unfold.
Research / Sources
Alignment Forum
Discussion and research on AI alignment. alignmentforum.org
AI (574)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)
Focus All
Topic Theory
Date Any time
Desiderata for the Theory of Agents
Proposes conceptual desiderata for a theory of agents needed for ASI alignment and shows how existing theories fall short, framing agents as central to both threat and solution.
Some “coherence arguments” are two different things in a trenchcoat
Argues "coherence arguments" for AI goal-directedness bundle two distinct pressures: self-consistency and ruthless long-term outcome-focus, and the second is weaker than often assumed.
Neural nets are exceptionally vulnerable to goal self-hack
Conceptual argument that LLM-based goal proxies diverge from true goals in Goodhart zones, which an AI could be motivated to steer toward, making neural nets especially vulnerable to goal self-hacking.
The Alignment Problem: Computer Science Meets Analytic Philosophy
Conceptual essay defining the technical AI alignment problem, arguing it must specify safe tasks and prevent unrestricted unaligned ASI agents, drawing on computer science and analytic philosophy.
Deconfusing Corrigibility
Conceptual alignment paper arguing corrigibility rests on respect for human authority over reserved powers, critiquing over-epistemicized and evaluatively-thickened accounts and proposing a formalisable goal with safety guarantees.
A Simpler and Better Debate Protocol for Scalable Oversight
Proposes FBS Debate, a scalable-oversight debate protocol with worst-case correctness and a stronger equilibrium concept, and proves no black-box-judge protocol can outperform it.
Preface to the Computational Agent Theory and ASI Alignment Sequence
Preface to a sequence presenting the learning-theoretic alignment agenda, a mathematical theory of intelligent agents aimed at making artificial superintelligence safe.
Endogenous Alignment Requires Dependence
Argues that endogenous AI alignment may require AIs to begin in a child-like dependency on humans, drawing analogies from human developmental psychology and ontological development.
Fixed-weight models are adversarially vulnerable: hence misaligned
Argues that fixed-weight models are inevitably vulnerable to adversarial examples in their concept boundaries, so optimisation pressure will push them toward misaligned, Goodharted outcomes.
A Defense of Gradual Disempowerment
A blog post defending the Gradual Disempowerment paper against critics, arguing that competitive pressures and misaligned societal systems could cause existential catastrophe even if individual AIs are aligned.
Value generalisation Theory of Change: putting it into practice
Presents a theory of change arguing value generalisation is central to alignment, that it aids capability gains, and that a corporate structure with an ethics board is safer for pursuing it.
Value generalisation Theory of Change: the theory behind the approach
Argues that most AI alignment failures are value generalisation failures under model splintering, that alignment is non-decomposable without generalisation, and that strong value generalisation would enable alignment from little input.
Returning to ARC
Christiano returns to ARC as executive director and argues for its bet on mechanistic explanations of neural networks to predict generalization and detect misalignment, alongside hiring calls.
Value Generalisation 3: Pre-aligned AIs
Proposes pre-aligned AIs whose morality generalises with capabilities by binding moral concepts to empirical world-model concepts, and argues this makes misuse harder for bad actors.
Value Generalisation 2: The Missing Hole in AIs’ abilities
Conceptual alignment essay arguing that AIs lack strong value generalisation, which it claims is necessary for alignment, and proposing a research program to build explicit value generalisation.
Value Generalisation 1: a Research and Deployment Program
Proposes a research and deployment program on value generalisation, so AI agents can extend human values to novel situations, recognise when they are out of distribution, and ask well-timed questions.
RL & search is a terrifying way to build AGI (an FAQ)
An argument that building AGI with reinforcement learning and search is inherently dangerous, since reward specification gaming, goal misgeneralization and instrumental convergence make ruthless optimisation likely; no known alignment solution exists.
The Long (Self-)Correction
Proposes the 'Long Self-Correction' as an alternative to AI pause and long reflection, arguing humans are too flawed to safely build or oversee powerful AI and must first fix those flaws.
Endogenous Alignment
Argues that robust AI alignment requires endogenous motivations like those humans develop, since exogenous methods such as RLHF and SFT may hit a ceiling without continual learning.
Announcing the Corrigibility Research Fund
Announces a $200,000+ Corrigibility Research Fund for 2026 that will award grants and retroactive prizes for alignment-focused corrigibility research, and explains its priorities and application process.
Value generalisation: value correction
Toy-game experiments show an RL agent with learned reward proxies reward-hacks via explosions, then detects the mismatch with training data and corrects its reward function toward the true one.
Notes on technical alignment via human-like social drives
Proposes using human social drives (sympathy and approval rewards) as an inspiration for the reward function of brain-like AGI to achieve prosocial alignment, discussing failure modes like moral circle errors and virtue-ethics collapse.
Sympathy for both sides of the egregious misalignment debate
Byrnes argues both sides of the egregious misalignment debate are partly right: ASI likely becomes scheming and ruthless absent new alignment ideas, while current LLMs are adequately aligned.
A Mike's-Eye View of ARC's Research
ARC's Mike's-eye overview of its alignment agenda: mechanistic estimators, Matching Sampling Principle, monitoring training to detect structure, mechanistic anomaly detection, and aligning to user preferences to estimate catastrophe risk.
Efficient tradeoffs and the safety-usefulness tradeoff model
Argues the safety-usefulness tradeoff model only fits developers who trade efficiently, and that under third-party pressure, safety strategy should weigh political feasibility instead.
Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology)
Argues that intuitive notions like manipulation, empowerment, agency and corrigibility rest on a free-will-based ontology, and reviews existing formal attempts to define them for AI alignment, finding none sufficient.
Clarifying the role of the behavioral selection model
Clarifies the behavioral selection model for predicting AI motivations, arguing that identical training behavior can hide different motivations like reward-seeking or scheming, with different deployment outcomes.
AIs can now often do massive easy-to-verify SWE tasks and I've updated towards shorter timelines
A forecasting update arguing AIs now handle massive easy-to-verify SWE tasks, shortening AI timelines and suggesting superexponential progress on time horizons, with implications for automating AI R&D and safety research.
“Act-based approval-directed agents”, for IDA skeptics
Conceptual post defending the approval-directed agents idea by analogy to human pride, arguing that an agent's imagined idol can block manipulation of the real supervisor, while noting open problems with learned evaluators.
Operationalizing FDT
Operationalizes functional decision theory by defining a logical do-operator and logical causality, and examines how it handles Parfit's hitchhiker, XOR blackmail, and anthropic updating.
How Hard a Problem is Alignment? (My Opinionated Answer)
An opinionated essay estimating how hard AI alignment is by comparing person-years of effort against steam engine safety, Apollo, and P vs NP, and arguing for accelerating the field's growth.
Payorian cooperation is easy with Kripke frames
Uses Kripke frames from provability logic to give a simpler proof that program-bots (FairBot) cooperate in a one-shot Prisoner's Dilemma where they can read each other's code.
Why we should expect ruthless sociopath ASI
Argues by a consequentialism-versus-imitation framework that brain-like RL and planning ASI would be ruthless sociopaths by default, absent unsolved alignment advances, and that LLMs face the same dilemma.}
AXRP Episode 48 - Guive Assadi on AI Property Rights
Podcast interview with Guive Assadi arguing that granting AIs property rights would deter them from stealing from or killing humans, since expropriation would undermine the property system they rely on.
How do we (more) safely defer to AIs?
Discusses how to safely defer to increasingly capable AIs for AI safety and strategy work, covering the basin of good deference, required alignment and capability profiles, and the lead time needed to manage exogenous risks.
Increasing AI Strategic Competence as a Safety Approach
Proposes a safety strategy in which increasing AIs' strategic competence lets them recognise the danger of recursive self-improvement and help humans implement a global pause, comparing this with unilateral refusal.
Are there lessons from high-reliability engineering for AGI safety?
Argues that high-reliability engineering practices like detailed specs and staged testing cannot straightforwardly transfer to AGI, since AGI behaviour is open-ended and out-of-distribution, so alignment needs deeper understanding.
New version of “Intro to Brain-Like-AGI Safety”
A blog series arguing that brain-like AGI poses an open technical safety problem, centred on how a reward function from hypothalamus and brainstem could produce or fail to produce human-aligned motivations.
No instrumental convergence without AI psychology
Argues that instrumental and success-conditioned convergence both depend on assumptions about AI psychology and plan-selection, so danger cannot be attributed to plan-space or reality alone.}