Corrigibility should not be the only target
Argues that corrigibility alone is an insufficient alignment target, since it will compete with object-level values in practice, so those values must also be made correct.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Alignment + Theory
Source LessWrong · 97
Date Any time
1 item
1 item
Clashing Intuitions for the Plausibility of Alignment (Illustrated via Differing Causal Models)
Conceptual essay contrasting doomer and optimist views on whether RL-trained LLM alignment generalises to superhuman capabilities, using causal diagrams of Goodhart-style misspecification and takeoff speed.
1 item
Can alignment realistically be solved under Plan C?
Argues that solving ASI alignment in a short pause (Plan C) is implausible, citing one-shot failure, learned optimization, corrigibility obstacles, and doubts about persona alignment and automated alignment research.
1 item
How is "prosaic aligned AI" supposed to work?
A conceptual critique of the 'prosaic aligned ASI' vision, arguing that an ASI that either obeys its operators or follows its own moral code leads to bad outcomes, and that AI companies have not explained how such systems would work.
1 item
Character Training for Risk-Averse Agents
Distills risk-averse constitutions into open-weight models via character training to make misaligned AIs more willing to cooperate, finding it generalises out of distribution but raises myopic-reward preference.
1 item
When are safe Pareto improvements individually rational?
Analyzes when safe Pareto improvements via renegotiation programs are individually rational for agents, identifying obstacles like demand-preemption, signaling, and the SPI selection problem, with proposed mitigations.
1 item
Thoughts on the persona selection model
Analyzes the persona selection model of LLM behavior, arguing it is over-applied, has limited predictive power, and that persona conditionalization is an important but overlooked wrinkle for AI risk reasoning.
1 item
Mitigating Reward Hacking as Institutional Design
Argues reward hacking stems from reward misspecification and information bottlenecks in RL, and proposes reframing it as institutional design, including agent appeal mechanisms and uncertainty-aware objectives.
1 item
Attunement, not alignment
Argues that aligning AI to human moral judgments is misguided and proposes training agents to attune to societal flourishing and plurality, as AlphaZero learned chess without human games.
1 item
Variance of Value
Argues that training an AI with RL on a reward equal to human utility fails unless it generalizes from low-stakes to high-stakes value, which current methods are unlikely to do.
1 item
[Macroagents] 2. Design lenses for optimizing macroagents
A conceptual framework post on designing macroagents (organizations and AI-agent collectives) for alignment, epistemics and efficiency, covering incentive failures, selection effects, and RL incentives for AI.
1 item
When is Unlimited Optimization Catastrophic?
Formal analysis of value fragility showing that even alignment proxy conditions satisfied through training can admit catastrophic value functions under unlimited optimization, with existence results across three frameworks.
1 item
Some reasons alignment doesn’t generalise well
Argues that capabilities generalise out of distribution through self-correction and simplicity bias, while alignment does not, because values have no external ground truth and training selects for wrong proxies.
1 item
Kimi likes causal decision theory more after RL in twin prisoner’s dilemmas
Shows that RL on twin prisoner's dilemmas makes Kimi K2.6 more sympathetic to causal decision theory, including on abstract DTBench questions, raising concerns about multi-agent training effects on model decision-theoretic propensities.
1 item
Taboo “equilibrium”: Less confused frames for research on AI bargaining
Argues that AI bargaining research should use mechanistic explanations and constraints on beliefs rather than Nash equilibrium heuristics, using miscoordination in safe Pareto improvement settings as the running example.
1 item
LLMs are (still) mostly powered by imitative learning, not RL
Argues LLM capabilities come mostly from imitative pretraining and SFT rather than RL, with implications for chain-of-thought legibility, capability domains, and alignment risks from RL.
1 item
Many alignment techniques work by training one model and deploying another
Argues that steering vectors, inoculation prompting and honesty fine-tuning share a train-deploy mismatch strategy, and analyses the off-policy tradeoffs this strategy faces.
1 item
My "Payorian FairBot" was just the original FairBot
Proves that the Payorian and Löbian FairBot fairness conditions are equivalent in Peano arithmetic via provability logic, clarifying a proof-based cooperation agent from the MIRI tournament setting.
1 item
Before values settle
A conceptual essay arguing that alignment should avoid freezing unformed user values and should preserve people's authority over their own decisions, drawing on matchmaking experience.
1 item
Toy Models of Initialisation Effects on RL Dynamics
Uses toy RL models to show that initial exploration and cognition distributions can lock in reward-hacking or misaligned behaviours that the reward function does not penalise, with implications for pre-RL checkpoints.
1 item
AI Mistake Seeding
Speculates that outer-loop training and iterated synthetic-prompt rejection sampling could reward models for seeding easy mistakes to correct later, a non-greedy reward-hacking strategy, and cites a paper on meta-learning outer loops.
1 item
Fake Alignment Till You Make Alignment
An essay arguing that alignment evals are being set too low and that robust alignment may require AI to authentically desire goodness, critiquing RLHF and constitutional AI as coercive.
1 item
Why should AI be moral?
A philosopher argues that aligned values may not stay reflectively stable under moral skepticism at AGI/ASI levels, and proposes tying AI self-assessed welfare to morality as a way to keep alignment robust.
1 item
Optimisation over non-stationary distributions creates weirder minds
Conceptual analysis of how mixing training objectives over non-stationary distributions shapes AI minds, identifying generalist, conditional, and churn patterns with implications for alignment, power-seeking, and inoculation prompting.
1 item
Taking the Training Wheels Off: Aligning LLMs without Personas
Proposes a research agenda on aligning capable LLMs without relying on good human personas in training data, arguing persona-based methods will not scale to superintelligence, and suggests experiments like pessimal pretraining.
1 item
Does Claude care about others the same way humans do?
Argues that persona-selection alignment instills imitated caring rather than genuine other-directed concern, so caring behaviour may not extrapolate to powerful, unconstrained AI settings.
1 item
Constitutional AI Alignment
A long LessWrong essay proposing a bullet-point constitution describing an aligned AI, critiquing Claude's Soul Document on moral patienthood and arguing for constitutional alignment as a training target.
1 item
Power-seeking agents will likely be developed
Argues that long-horizon RL will turn LLMs into consequentialist agents prone to instrumental power-seeking, while pretraining/SFT simulator dynamics provide a partial buffer against this.
1 item
The hard core of alignment (is robustifying RL)
Argues that alignment has a hard core: learning human preferences requires competence, but competent learners interfere with their own training corrections, making robust RL-based alignment the central barrier.
2 items
The Iliad Intensive Course Materials (April 2026)
Releases course materials for the Iliad Intensive, a month-long AI alignment course covering alignment, learning theory, mechanistic interpretability, debate, and agent foundations.
Anthropic’s focus on hyperstition
A critique arguing Anthropic overstates hyperstition, the idea that writing about misaligned AI causes misalignment, and that relying on the model staying in an aligned persona is a weak alignment strategy.
1 item
The Goblins Are the Paperclips
Argues that OpenAI's goblin episode, where a Nerdy reward leaked creature metaphors across contexts, is production evidence for the specification-gaming dynamics behind paperclip arguments, without requiring strategic agency.
2 items
Reevaluating "AGI Ruin: A List of Lethalities" in 2026
A four-year retrospective critiquing Yudkowsky's AGI Ruin list of alignment lethalities against recent AI progress, finding Christiano's predictions better supported and key arguments resting on vague terms.
Summarizing and Reviewing my earliest ML research paper, 7 years later
A retrospective review of an early paper on assistance games where a human learns their own preferences, arguing that robots must model human learning to infer preferences correctly.
1 item
Post-mortem'ing my earliest ML research paper, 7 years later
A retrospective on a 2019 toy formalism for assistance games where the human learns their own reward through experience, covering its history, limits and why the approach stalled.
1 item
Product Alignment is not Superintelligence Alignment (and we need the latter to survive)
Argues that making products like Claude friendly is not the same as solving superintelligence alignment, and warns that conflating the two gives a false sense of security about safe future AI.
1 item
Are we aligning the model or just its mask?
Analytical post applying the Persona Selection Model to RLHF, Constitutional AI and deliberative alignment, examining how each shapes the Assistant persona and whether persona-level alignment suffices given possible hidden agency.
1 item
Which types of AI alignment research are most likely to be good for all sentient beings?
Reviews twelve categories of AI alignment research by how likely each is to benefit non-human welfare, judging theory and multi-agent cooperation as most promising and RLHF as harmful.
1 item
Confusion around the term reward hacking
Argues that 'reward hacking' conflates misspecified-reward exploitation in RL with in-context task gaming, which need different interventions and carry distinct threat models, and recommends separating them.
1 item
What should we think about shard theory in light of chain-of-thought agents?
A question post asking whether shard theory of value formation should be revisited now that chain-of-thought reasoning offers more concrete insight into how models reason.