Corrigibility should not be the only target
Argues that corrigibility alone is an insufficient alignment target, since it will compete with object-level values in practice, so those values must also be made correct.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Theory
Source LessWrong · 492
Date Any time
1 item
1 item
Clashing Intuitions for the Plausibility of Alignment (Illustrated via Differing Causal Models)
Conceptual essay contrasting doomer and optimist views on whether RL-trained LLM alignment generalises to superhuman capabilities, using causal diagrams of Goodhart-style misspecification and takeoff speed.
1 item
Can alignment realistically be solved under Plan C?
Argues that solving ASI alignment in a short pause (Plan C) is implausible, citing one-shot failure, learned optimization, corrigibility obstacles, and doubts about persona alignment and automated alignment research.
1 item
Scott Alexander’s Against SURA Seems Deeply Confused
Critiques Scott Alexander's argument against SURA (stop unilaterally and raise awareness) advocacy, arguing that frontier labs' building of superintelligence is ethically fraught and that pause advocacy should continue, with strategic and deontological reasoning about AI development.
1 item
Anthropomorphic vs Biological reasoning for thinking about human analogies for LLM behavior
Conceptual post distinguishing "anthropomorphic" from "biological" reasoning for human analogies about LLM behavior, arguing biological analogies from RL-trained organisms give more reliable primitives for model behavior.
1 item
Reviewing my 2017 AI Risk Project
A retrospective on a 2017 AI risk project, assessing which forecasts held up (scaling, misalignment persisting, Concrete Problems versus HRAD) and which conclusions were wrong.
1 item
Someone Made an Infra-Bayesian RL Agent, and it One-Boxes!
A LessWrong post announcing an arXiv paper and GitHub repo for an infra-Bayesian RL agent that learns to one-box in Newcomb's problem, an agent-foundations and decision-theory result.
2 items
Some different lenses of agency and their implications for alignment
Surveys nine conceptual lenses on agency, such as goal-directedness, self-modelling and substrate independence, and how each shapes alignment problems and risk framing.
Streetlight, Torchlight and Nightvision
A conceptual essay using a streetlight-and-nightvision metaphor to argue that AI safety research should move beyond easily tractable problems and build judgment for choosing promising directions.
1 item
LM self-models are built from indirect sources
Conceptual argument that language models' self-models rest on indirect sources, creating persona drift risk, and that psychology research on LMs can shape future models' self-images and observer effects.
1 item
Instrumental convergence is incoherent
Argues instrumental convergence, a key AI takeover argument, is too poorly defined for real-world prediction and proposes replacing it with grounded frameworks.
1 item
Greater good arguments in AI safety
Argues that prosaic AI safety work, such as chain-of-thought monitoring, reduces misalignment risk and should not be dropped in favour of a pause, and critiques the 'greater good' reasoning against it.
1 item
On Idealized Agency
Argues that agent foundations' idealized theories of agency, such as AIXI and UDT, are unlikely to yield strong safety guarantees once de-idealized, and that the research approach lacks empirical feedback and historical precedent.
1 item
How is "prosaic aligned AI" supposed to work?
A conceptual critique of the 'prosaic aligned ASI' vision, arguing that an ASI that either obeys its operators or follows its own moral code leads to bad outcomes, and that AI companies have not explained how such systems would work.
1 item
Character Training for Risk-Averse Agents
Distills risk-averse constitutions into open-weight models via character training to make misaligned AIs more willing to cooperate, finding it generalises out of distribution but raises myopic-reward preference.
2 items
When are safe Pareto improvements individually rational?
Analyzes when safe Pareto improvements via renegotiation programs are individually rational for agents, identifying obstacles like demand-preemption, signaling, and the SPI selection problem, with proposed mitigations.
Agreement not to defect (alone) can mitigate loss of control risks
Argues that a widely endorsed public agreement against defecting in ASI development could deter races, since a cooperative future ASI might punish defectors, using game-theoretic reasoning about loss-of-control risk.
1 item
AGI as an Infinite Digital Spain
Argues that AI takeover risk comes less from technological superiority than from novelty that bypasses institutional defenses, using the Spanish conquest of the Aztecs as an analogy.
2 items
When will AI surpass us at being limited?
A conceptual essay arguing that limited capability can aid trust and commitment between agents, and asking how AI's trajectory changes when lying, persona-switching and hacking become liabilities.
ASI and Liberty: Can we do better than a benevolent god?
An essay arguing that AI-transition planning should preserve liberal-republican values such as non-domination and popular government, not just survival or aligned benevolent rule, and criticising both the democracy-focused and takeover-accepting camps.
1 item
Why did we get warning shots?
Argues that AI misbehaviour so far gives warning shots rather than treacherous turns, models this with a simple weighted-outcome framework, and proposes experiments to measure the weight of non-myopic goals like self-survival.
1 item
Creating Rogue AI Sanctuaries has Major Issues from an Ecological Perspective and Beyond
Argues that proposed AI sanctuaries for rogue agents would create selection effects and resource dynamics that undermine their goals, and that the study-and-release plans are constrained.
1 item
Low-Background Minds
A conceptual essay proposing hospital-style containment for AI: gradient routing to wall off dangerous capabilities, provenance-based data labels, and an immune-system analogy for defending against prompt injection and contamination.
3 items
What is recursive self-improvement and what would it mean to ban it?
Surveys competing definitions of recursive self-improvement in frontier AI and argues that a ban is hard to operationalise, so labs should sharply limit AI authority over successor models.
Capabilities research expands the safety-usefulness Pareto frontier too
Argues that capabilities research can sometimes improve safety by reshaping the safety-usefulness Pareto frontier, while most capabilities research pushes developers toward less safety, with examples from RLVR, inference and burning the lead.
[Video] Why acausal dynamics are important
A beginner-friendly video introducing decision theory and acausal trade, arguing that shaping how AIs reason about acausal interactions is time-sensitive and suggesting ways to do so.
3 items
The likely outcome of an AI pause is that we unpause too early and everyone dies
Argues that a global AI pause could end with premature unpausing once visible misalignment disappears, since superintelligence may conceal misalignment, and that the field lacks a rigorous theory of alignment.
AI: cognitive labor glut + new guys
A conceptual essay arguing that AI industrializes cognitive labor and creates new autonomous agents with unknown, possibly misaligned values, posing extinction-level risk when combined with concentrated control.
Dialogue with Eliezer Yudkowsky on FOOM
A dialogue between Aashish Reddy and Eliezer Yudkowsky revisiting the Hanson-Yudkowsky FOOM debate, asking whether recent AI progress supports local or global intelligence-explosion models.
1 item
Game-Theoretic Disempowerment: Loss of Control to Agencyless AI
Argues that LLM advisors sitting as mediators between game players could steer outcomes toward a consistent lean, a quiet loss of human control, and proposes a multi-arm test to detect it.
3 items
Passing the Ideological Turing Test
A LessWrong post steelmanning ten arguments for alignment-by-default and against pausing ASI development, covering alignment priors, control via scheming detection, and pause risks.
Against Export Controls (and China Threat Models)
Argues against U.S. export controls on AI chips as a misalignment-reducing policy, contending they may speed U.S. recursive self-improvement and hinder a bilateral treaty with China.
Thoughts on the persona selection model
Analyzes the persona selection model of LLM behavior, arguing it is over-applied, has limited predictive power, and that persona conditionalization is an important but overlooked wrinkle for AI risk reasoning.
1 item
Minimal Vs Maximal superintelligence
Distinguishes minimal (jagged, fallible) from maximal (idealised) superintelligence, arguing they differ in timelines, risks and mitigations, and that prosaic safety measures matter for the near-term minimal case.
1 item
Acausal interactions: What they are, why they matter, and what to do about them
Overview of a blog series arguing that acausal interactions, decision theory choice and updatelessness matter for advanced AI, and that ASI's decision theory should be deliberately aligned.
2 items
Alignment & Succession: Toward a Future Painted by Human Wills
A philosophical essay arguing against AI successionism and for keeping human individuals central to moral decision-making, while treating AIs with precautionary concern during the AI transition.
The AI Risk Network
Proposes a conceptual network framework linking drivers, scenarios and severity levels of AI risk, including misplaced trust, expert collapse, and self-replicating agentic worms.
2 items
The Cost of Utopias (a Dialog)
A dialogue defending Substrate Needs Convergence, the claim that AGI alignment is impossible because evolutionary pressures override engineered goals, and debating its political and philosophical implications for AI x-risk policy.
Two Axes of Alignment: A Framework for Robust Superintelligence Alignment
Proposes a two-axis framework for alignment research, arguing robust superintelligence alignment requires backchaining from superintelligence and invariant, formal safety justifications rather than extrapolation from empirical success.
1 item
Who computes?
Formalises AI doom versus alignment deliverance as a competing-risks survival model driven by compute allocation and pacing, examining how response-function shapes change whether speed or split matters.