Notes on Take-Off Speeds and RSI
Course notes reviewing AI take-off speed debates (Christiano vs. Yudkowsky) and recursive self-improvement, framing how fast advanced AI risk could unfold.
Research / Topics
Theory & Foundations
Agent foundations, decision theory and corrigibility.
Focus All
Source All
Date Any time
Is Recursive Self-Improvement Safe?
Argues that frontier AI makes recursive self-improvement plausible, that no clear frameworks predict its outcomes, and offers a game-theoretic analysis of whether to start RSI when rivals might too.
Corrigibility should not be the only target
Argues that corrigibility alone is an insufficient alignment target, since it will compete with object-level values in practice, so those values must also be made correct.
Clashing Intuitions for the Plausibility of Alignment (Illustrated via Differing Causal Models)
Conceptual essay contrasting doomer and optimist views on whether RL-trained LLM alignment generalises to superhuman capabilities, using causal diagrams of Goodhart-style misspecification and takeoff speed.
Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback
Proposes ZO-LBPG, a reward-model-free zeroth-order log-barrier method for safe RLHF that keeps every training policy feasible, with convergence and sample-complexity guarantees plus small experiments.
SAVeR$^2$: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards
Proposes SAVeR² for safety-aligning large reasoning models using four fine-grained verifiable rewards and reasoning-preserving gradient projection, reporting better safety and lower over-refusal without losing reasoning.
Goal Revelation as a Bridge Between Information-Theoretic Empowerment and Capability-Based Human Power
Links information-theoretic empowerment and capability-based human power via a goal-revelation bridge, showing empowerment-maximising or revelation-maximising assistants can fail, and proposing a satisfaction floor for monitoring retained human power.
Adapting the Process of Adaptation: Darwinian, Lamarckian, and Agent-Directed Evolution in AI
Theoretical framework comparing Darwinian, Lamarckian, and agent-directed AI evolution; finds misspecified evaluation targets accelerate convergence to wrong behavior, risking persistent errors in recursive self-improvement.
Desiderata for the Theory of Agents
Proposes conceptual desiderata for a theory of agents needed for ASI alignment and shows how existing theories fall short, framing agents as central to both threat and solution.
A Definition of Catastrophic Performance
Gives a formal definition of catastrophic performance from AI misalignment, using baseline and orthogonal reference values to separate catastrophic outcomes from merely suboptimal ones.
Can alignment realistically be solved under Plan C?
Argues that solving ASI alignment in a short pause (Plan C) is implausible, citing one-shot failure, learned optimization, corrigibility obstacles, and doubts about persona alignment and automated alignment research.
Scott Alexander’s Against SURA Seems Deeply Confused
Critiques Scott Alexander's argument against SURA (stop unilaterally and raise awareness) advocacy, arguing that frontier labs' building of superintelligence is ethically fraught and that pause advocacy should continue, with strategic and deontological reasoning about AI development.
Some “coherence arguments” are two different things in a trenchcoat
Argues "coherence arguments" for AI goal-directedness bundle two distinct pressures: self-consistency and ruthless long-term outcome-focus, and the second is weaker than often assumed.
Anthropomorphic vs Biological reasoning for thinking about human analogies for LLM behavior
Conceptual post distinguishing "anthropomorphic" from "biological" reasoning for human analogies about LLM behavior, arguing biological analogies from RL-trained organisms give more reliable primitives for model behavior.
Reviewing my 2017 AI Risk Project
A retrospective on a 2017 AI risk project, assessing which forecasts held up (scaling, misalignment persisting, Concrete Problems versus HRAD) and which conclusions were wrong.
Someone Made an Infra-Bayesian RL Agent, and it One-Boxes!
A LessWrong post announcing an arXiv paper and GitHub repo for an infra-Bayesian RL agent that learns to one-box in Newcomb's problem, an agent-foundations and decision-theory result.
Efficient Scalable Oversight via Rewinding
Introduces rewindable interactive proofs, letting a weak verifier rewind AI provers, giving a cryptography-free oversight protocol that does not assume a truthful prover, with an MNIST experiment.
Loss is Not Behavior: A Unified Output-Space Analysis of Gradient-Based Machine Unlearning
Derives closed-form output-space gaps for gradient-based unlearning in linear regression, extended to nonlinear models, and proposes leverage diagnostics for auditing forget sets.
Some different lenses of agency and their implications for alignment
Surveys nine conceptual lenses on agency, such as goal-directedness, self-modelling and substrate independence, and how each shapes alignment problems and risk framing.
Streetlight, Torchlight and Nightvision
A conceptual essay using a streetlight-and-nightvision metaphor to argue that AI safety research should move beyond easily tractable problems and build judgment for choosing promising directions.
A basic case for accelerating conceptual reasoning capabilities
Redwood argues that AI's conceptual reasoning, needed for alignment and philosophy-like work, should be differentially accelerated and is underinvested in, and promises a follow-up agenda.
LM self-models are built from indirect sources
Conceptual argument that language models' self-models rest on indirect sources, creating persona drift risk, and that psychology research on LMs can shape future models' self-images and observer effects.
Responsibility Boundaries for Agentic Systems: A Comparative Evaluation of Probabilistic Agentic Programming
Proposes Probabilistic Agentic Programming, with a mediation kernel enforcing authority, approval and resource checks before LLM agent tool calls, proves conditional safety, and reports a small comparative study of observed task outcomes.
Instrumental convergence is incoherent
Argues instrumental convergence, a key AI takeover argument, is too poorly defined for real-world prediction and proposes replacing it with grounded frameworks.
Aligning Large Language Models with Human Feedback: Mathematical Foundations and Algorithm Design
A tutorial surveying the mathematical foundations and algorithms for aligning LLMs with human feedback, including SFT, RLHF, DPO, and links to inverse RL and discrete choice models.
Greater good arguments in AI safety
Argues that prosaic AI safety work, such as chain-of-thought monitoring, reduces misalignment risk and should not be dropped in favour of a pause, and critiques the 'greater good' reasoning against it.
What Lies Beyond Perception: A Partition Theory of the Limits of AI Evaluation and Oversight
Develops a partition theory showing behavioural evaluations can only certify a floor on a model's discrimination, so detecting concealed capabilities or sandbagging via evaluation is fundamentally limited.
On Idealized Agency
Argues that agent foundations' idealized theories of agency, such as AIXI and UDT, are unlikely to yield strong safety guarantees once de-idealized, and that the research approach lacks empirical feedback and historical precedent.
Neural nets are exceptionally vulnerable to goal self-hack
Conceptual argument that LLM-based goal proxies diverge from true goals in Goodhart zones, which an AI could be motivated to steer toward, making neural nets especially vulnerable to goal self-hacking.
Poisoning-Based Backdoors Are Not Cryptographic
Proves that poisoning-based backdoors cannot be made cryptographically asymmetric: any verifier poisoning can install is reconstructible by a white-box defender, confirmed on instruction-tuned LLMs.
Fundamental Impossibility Results in AI Safety: Trading Off between Desiderata
Formalises AI safety as a three-player game and proves a fundamental tradeoff: advice from an AI that helps a bounded defender while remaining public cannot avoid costs, and manipulating shared information to benefit only the defender is also examined.
How is "prosaic aligned AI" supposed to work?
A conceptual critique of the 'prosaic aligned ASI' vision, arguing that an ASI that either obeys its operators or follows its own moral code leads to bad outcomes, and that AI companies have not explained how such systems would work.
The Alignment Problem: Computer Science Meets Analytic Philosophy
Conceptual essay defining the technical AI alignment problem, arguing it must specify safe tasks and prevent unrestricted unaligned ASI agents, drawing on computer science and analytic philosophy.
Deconfusing Corrigibility
Conceptual alignment paper arguing corrigibility rests on respect for human authority over reserved powers, critiquing over-epistemicized and evaluatively-thickened accounts and proposing a formalisable goal with safety guarantees.
Character Training for Risk-Averse Agents
Distills risk-averse constitutions into open-weight models via character training to make misaligned AIs more willing to cooperate, finding it generalises out of distribution but raises myopic-reward preference.
A Simpler and Better Debate Protocol for Scalable Oversight
Proposes FBS Debate, a scalable-oversight debate protocol with worst-case correctness and a stronger equilibrium concept, and proves no black-box-judge protocol can outperform it.
Preface to the Computational Agent Theory and ASI Alignment Sequence
Preface to a sequence presenting the learning-theoretic alignment agenda, a mathematical theory of intelligent agents aimed at making artificial superintelligence safe.
The Self-Improvement Problem
Formal impossibility results show that no terminating gate or search can reliably accept only genuine self-improvements of agents in unbounded settings, while bounded settings with finite budgets can recover guarantees.
When are safe Pareto improvements individually rational?
Analyzes when safe Pareto improvements via renegotiation programs are individually rational for agents, identifying obstacles like demand-preemption, signaling, and the SPI selection problem, with proposed mitigations.
Agreement not to defect (alone) can mitigate loss of control risks
Argues that a widely endorsed public agreement against defecting in ASI development could deter races, since a cooperative future ASI might punish defectors, using game-theoretic reasoning about loss-of-control risk.