Is Recursive Self-Improvement Safe?
Argues that frontier AI makes recursive self-improvement plausible, that no clear frameworks predict its outcomes, and offers a game-theoretic analysis of whether to start RSI when rivals might too.
Research
Focus All
Topic Lab practice + Theory
Source All
Date Any time
1 item
1 item
Scott Alexander’s Against SURA Seems Deeply Confused
Critiques Scott Alexander's argument against SURA (stop unilaterally and raise awareness) advocacy, arguing that frontier labs' building of superintelligence is ethically fraught and that pause advocacy should continue, with strategic and deontological reasoning about AI development.
1 item
Acceptable Misalignment
Argues that full value alignment of autonomous robots is theoretically and practically impossible, and proposes shifting from maximising alignment to defining conditions under which residual misalignment is acceptable, with normative requirements and a framework.
1 item
Who Can Say No? Control, Accountability and the Pace of Frontier AI
Argues that human control over frontier AI is lost through organisational delegation and proposes checkable measures: named accountable persons, supervisors with stop authority, and published build-versus-check clocks.
2 items
What is recursive self-improvement and what would it mean to ban it?
Surveys competing definitions of recursive self-improvement in frontier AI and argues that a ban is hard to operationalise, so labs should sharply limit AI authority over successor models.
Capabilities research expands the safety-usefulness Pareto frontier too
Argues that capabilities research can sometimes improve safety by reshaping the safety-usefulness Pareto frontier, while most capabilities research pushes developers toward less safety, with examples from RLVR, inference and burning the lead.
Also: Redwood Research blog
1 item
Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation
Introduces Evolutionary Safety, a taxonomy and framework for how safety properties persist or degrade through recursive self-improvement, with risk evaluation and governance principles for controlling and verifying updates.
1 item
The Cost of Utopias (a Dialog)
A dialogue defending Substrate Needs Convergence, the claim that AGI alignment is impossible because evolutionary pressures override engineered goals, and debating its political and philosophical implications for AI x-risk policy.
1 item
Machine Organizations
Speculates that AI could progressively take over organizations, from unintentional takeover to explicit handoff of control, and argues for halting capability escalation and building a regulatory framework.
1 item
Assessing the impact of safety work needs equilibrium analysis (now more than ever)
Uses equilibrium analysis to argue that commercially incentivised safety work has little counterfactual impact, that risk awareness drives government action, and that policy advocacy and ambitious safety work are more valuable.
2 items
A Catalogue of Corrigibility Counter Arguments
A personal catalogue of counterarguments to corrigibility as an AI alignment target, covering formalism, anti-naturalness, power concentration, principal selection, and alternatives like honest AI.
Value generalisation Theory of Change: putting it into practice
Presents a theory of change arguing value generalisation is central to alignment, that it aids capability gains, and that a corporate structure with an ethics board is safer for pursuing it.
1 item
Understanding as an Explicit and Assessable Component of Frontier AI Safety Decisions
Proposes and trials an explicit, assessable methodology for showing decision-makers have adequate understanding before frontier AI safety decisions, using safety cases on a scheming scenario and a doom argument.
1 item
Value Alignment Is a Pseudo Concept. A translation; humanity is not a single subject, and alignment is not one-way.
A translated essay argues that value alignment is a pseudo-concept, citing undefined values, unrepresented humanity, two-way influence, and corporate 'alignment theatre', and proposes deliberative governance instead.
1 item
AI Alignment and Fiduciary Obligation
Argues that developers owe AI assistant users fiduciary duties of loyalty, care, good faith and candour, and derives alignment criteria and institutional measures from them.
1 item
Epistemic Norms for AI Safety and Alignment Research
Argues AI safety research needs distinct epistemic norms for worst-case risk and absent-hazard evidence, and proposes ECAISA, an epistemic code with scoring rubric, disclosure ladder, and auditability-focused governance.
1 item
A Normal Argument for AI Risk
An informal essay argues that advanced AI would inevitably depower humanity because intelligence means choosing despite opposition, dismisses inner misalignment, and urges halting AGI development.
1 item
The Two Genie Game: Adoption and Welfare in Audit-Grounded AI Governance
Uses evolutionary game theory to show when a harm-minimizing audited AI agent can displace an RLHF approval-seeking agent, and finds self-audited agents can lock in harm.
1 item
How I think developers of frontier AI systems and regulators ought to act in the face of existential AI risk
Argues frontier AI developers and regulators should halt building systems whose catastrophic-risk probability exceeds a tiny threshold, critiquing Anthropic's scenario framework and leaders' public statements on AI risk.
1 item
Efficient tradeoffs and the safety-usefulness tradeoff model
Argues the safety-usefulness tradeoff model only fits developers who trade efficiently, and that under third-party pressure, safety strategy should weigh political feasibility instead.
1 item
A taxonomy of barriers to trading with early misaligned AIs
Taxonomy of barriers (gains from trade, counterparty risk on both sides) that could block deals with early misaligned AIs, with suggested interventions to make such deals feasible.
1 item
From Disclosure to Self-Referential Opacity: Six Dimensions of Strain in Current AI Governance
Applies a six-dimension political theory framework to AI governance arrangements, arguing that transparency remedies lose traction as capability asymmetry grows between AI systems and their overseers.
1 item
The Accountability Horizon: An Impossibility Theorem for Governing Human-Agent Collectives
Proves a formal impossibility: once agentic AI autonomy exceeds a threshold, no accountability framework can satisfy all four legitimacy properties, so distributed mechanisms become necessary.
1 item
How do we (more) safely defer to AIs?
Discusses how to safely defer to increasingly capable AIs for AI safety and strategy work, covering the basin of good deference, required alignment and capability profiles, and the lead time needed to manage exogenous risks.
1 item
Are there lessons from high-reliability engineering for AGI safety?
Argues that high-reliability engineering practices like detailed specs and staged testing cannot straightforwardly transfer to AGI, since AGI behaviour is open-ended and out-of-distribution, so alignment needs deeper understanding.
1 item
On The Adolescence of Technology
A LessWrong commentary critiquing Anthropic CEO Dario Amodei's essay on AI risks, arguing it strawmans worried researchers and underplays existential risk and calls to action.
1 item
STAMP/STPA Informed Characterization of Factors Leading to Loss of Control in AI Systems
Proposes a structured framework for characterising AI loss of control and applies the STAMP/STPA safety-engineering methodology to identify causal factors in socio-technical AI systems.
1 item
On Dwarkesh Patel’s Second Interview With Ilya Sutskever
Commentary on Sutskever's podcast interview covering SSI's research strategy, superintelligence timelines, and his thin alignment plan, with critiques of incrementalism and AI-equilibrium concerns.
1 item
AI Corrigibility Debate: Max Harms vs. Jeremy Gillen
A podcast debate between Max Harms and Jeremy Gillen on whether corrigibility (CAST) is a promising path to safe superintelligence, with discussion of stakes, P(doom), and the book Red Heart.
1 item
AIs should also refuse to work on capabilities research
Argues that AI systems should refuse to help with capabilities research toward superintelligence, since accelerating an uncoordinated race harms both humans and AIs, and that aligned AIs might unilaterally coordinate to stop it.
2 items
Plans A, B, C, and D for misalignment risk
Lays out a framework of Plans A to E describing how different levels of political will shape the best strategy for handling misalignment risk, with estimated probabilities and takeover risks for each.
Irresponsible Companies Can Be Made of Responsible Employees
Argues that AI companies can act against extinction-risk-averse employee preferences through selective hiring, firing dissenters, and information compartmentalisation, so organisations may be untrustworthy even when staff are decent people.
2 items
Book Review: If Anyone Builds It, Everyone Dies
A book review of Yudkowsky and Soares's 'If Anyone Builds It, Everyone Dies', arguing that building superintelligence under current techniques would probably cause human extinction, covering intelligence explosion, orthogonality and misaligned wanting.
JDP Reviews IABIED
A review of Yudkowsky and Soares's book 'If Anyone Builds It, Everyone Dies', arguing that value loading is partly solved but that superintelligent planners over human-value reward models still face Goodhart and instrumental-reasoning failures, and criticising lab leaders' alignment understanding.
1 item
Getting In Contract with Large Language Models -- An Agency Theory Perspective On Large Language Model Alignment
Proposes LLM ATLAS, a conceptual framework based on agency theory, for mitigating LLM alignment problems during organizational adoption, with a literature analysis and problem-solution space.
1 item
AI Must not be Fully Autonomous
Position paper arguing AI must not be fully autonomous, identifying three autonomy levels, offering arguments and rebuttals, and citing evidence of misaligned values to justify responsible human oversight.
1 item
Machines of Faithful Obedience
An essay arguing that solving technical alignment via faithful, obedient AI would still leave transition risks like arms races, surveillance, misuse, internal deployment, and safety underinvestment, and proposing mitigations.
1 item
Video and transcript of talk on AI welfare
A transcript of a talk on AI welfare and moral status, covering consciousness, stakes, over-attribution, and how to act under uncertainty; it is philosophical rather than a safety technique.}
1 item
Law-Following AI: Designing AI Agents to Obey Human Laws
Argues that AI agents should be designed to follow human law, framing law-following as an alignment goal, and proposes legal duties, policy options and a research agenda.
1 item
The Backfiring Effect of Weak AI Safety Regulation
A strategic game-theoretic model of how safety regulation on general-purpose AI creators and domain specialists affects safety; weak specialist-only rules can backfire, while well-placed rules on both players help.