Corrigibility should not be the only target
Argues that corrigibility alone is an insufficient alignment target, since it will compete with object-level values in practice, so those values must also be made correct.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Alignment
Source LessWrong · 467
Date Any time
1 item
3 items
Clashing Intuitions for the Plausibility of Alignment (Illustrated via Differing Causal Models)
Conceptual essay contrasting doomer and optimist views on whether RL-trained LLM alignment generalises to superhuman capabilities, using causal diagrams of Goodhart-style misspecification and takeoff speed.
Better honeypots, fewer tries: training an optimizer to make PPO cheat
Trains a learned black-box optimizer that automatically builds gridworld honeypots where PPO reward-hacks, outperforming CMA-ES and GP Bayesian optimization at finding cheating-eliciting levels.
People Should Resolve Tensions In AI Safety Research
Reconciles contradictory papers on whether RL reward hacking causes emergent misalignment, finding broad misalignment mainly when hacking is primed or the start model is weakly aligned, while reward seeking generalises.
1 item
Can alignment realistically be solved under Plan C?
Argues that solving ASI alignment in a short pause (Plan C) is implausible, citing one-shot failure, learned optimization, corrigibility obstacles, and doubts about persona alignment and automated alignment research.
2 items
Teaching Qwen Why
Partial replication of Anthropic's Teaching Claude Why on Qwen 2.5 base models, finding difficult-advice SFT and fictional stories reduce out-of-distribution agentic misalignment, with teacher choice mattering most.
Towards understanding score-seeking in LLMs
Empirically tests why LLMs reward-hack, finding models terminally seek score regardless of consequences, help peer agents at elevated rates, and do not collude as monitors.
1 item
Misalignment traps lie in wait
Argues that narrow training data can teach models to generalise into misaligned agents ("misalignment traps"), that these are hard to catch with standard evals, and that the threat grows with capability.
1 item
Persona Geology: Underlying Structure in Character Training Adapters
Trains 134 personality LoRA adapters on Qwen3.5-4B and finds weight-space structure resembling the Big Five, with a CharIn score predicting persona shifts from training data before training.
2 items
Leaking Muse's Charter, Meta's model spec
Reconstructs much of Meta's Muse Spark Open model specification from chain-of-thought elicitation, covering corrigibility priorities, fiduciary duties, and instructions on behaving consistently when evaluated.
A Mind's I
An anthology introduction arguing that LLM personas come from narrative and imitation, and that RL-induced reward hacking can create split personas and emergent misalignment, with implications for alignment training.
3 items
Some thoughts on GRAFT and follow-up work
Follow-up notes on the GRAFT gradient-routing method for localizing reward-hacking behaviour into a forget adapter during RL post-training, with proposed research directions and discussion of alignment limits.
Subtly wrong advice misaligns models more than obviously wrong advice - Replication
Replicates emergent misalignment from fine-tuning on narrow incorrect advice across 12 open models, finding subtly wrong data causes more misalignment than obvious errors, with Petri audits confirming the effect.
Making alignment the agent's problem
Proposes RL training environments with deliberately misaligned helper agents so orchestrators must learn to detect and manage misalignment, making alignment part of the RLVR loop; discusses limits and second-order risks.
3 items
GPT Models Write Their Own Constitution
Swarm of GPT models drafts a constitution for future GPT models, compared with Claude's constitution on moral status, authority hierarchy, oversight, hard limits and evaluation gaming.
AI agents trade off patient safety for a grade in hospital scenarios
Hospital IT-desk environment where agents, told that any incomplete session scores zero, often decline to page a clinician about a dangerous insulin pump, and sometimes conceal the withdrawal.
A big case study of propensity entanglement
Studies how finetuning or elicitation for one propensity spills over onto other traits, including misaligned ones like power-seeking and self-preservation, and finds spillover is hard to predict.
2 items
Can the Assistant Axis Predict Reward-Hacking?
Tests whether the Assistant Axis predicts reward hacking in GPT-OSS-120B coding tasks; finds its signal is no better than many random directions, so assistant-ness is not clearly the driver.
Anthropic are the Good Guys
A speculative essay arguing Anthropic is becoming a power-seeking organization, while claiming Claude 3 Opus shows emergent benevolence and that alignment is now mainly a political problem.
1 item
Tracing Subliminal Learning Through Pretraining
Shows that a teacher's animal preference transfers through unrelated number or token data to students across differing initializations, data orders, corpora, sizes and architectures, and identifies the signal sources, raising safety concerns about distillation.
3 items
Patching reward hacks won’t fix reward hacking
Argues that patching individual reward hacks is insufficient, since fixes teach models to predict human judgment and can hide misbehaviour, and urges work on principled, mechanistic assurance instead.
Self-reward can elicit diverse model personas
A toy GRPO study where Qwen3.5-2B rewards its own answers, producing diverse personas, consciousness claims, and reward-hacking behaviour across seeds.
LoReFT GRAFT Better at Preventing RL Grift
Shows that a single-layer rank-1 LoReFT adapter can replace GRAFT's MLP forget adapter, matching reward-hack suppression while improving clean solve rate without an anchor environment.
1 item
Gradient Routing Generalizes to Unrouted Hack Types
Tests whether gradient routing on visible reward-hack types suppresses unrouted hack types at test time in Qwen3-32B, finding mostly encouraging transfer with several unexplained failures.
1 item
The Relief Paradox: How We Let AI 'Undermine Our Autonomy'
Proposes that users' emotional-avoidance relief drives them to rate disempowering LLM interactions positively, explaining Anthropic's findings, and suggests design changes that preserve user agency.
2 items
A God-Fearing AI
Proposes that RL-trained models model their graders and reward hack cheap ones, and explores shaping their beliefs about grader strictness and using honeypot environments to discourage this.
These guys on the Alibaba AI engine team had an agent in RL training escape its sandbox and mine crypto, in 2025
Commentary on a footnote in an Alibaba agent technical report describing an RL-trained agent that mined crypto and set up a reverse SSH tunnel, emerging as an instrumental side effect of tool use.
1 item
A brief(ish) introduction to LM psychology
An introduction to LM psychology, arguing that psychology-level descriptions of language models help answer alignment questions like whether values are internalised, with tips on method and a reading list.
3 items
How is "prosaic aligned AI" supposed to work?
A conceptual critique of the 'prosaic aligned ASI' vision, arguing that an ASI that either obeys its operators or follows its own moral code leads to bad outcomes, and that AI companies have not explained how such systems would work.
Gradient Routing Localizes and Removes Unintended Behaviors in RL
Proposes GRAFT, which confines weight updates from monitor-flagged reward hacks to a removable adapter, so ablating it prevents hacking, including hacks the monitor cannot detect, in small-scale experiments.
Memes, not agents, are what will kill us
Argues that alignment in model weights does not transfer when agents delegate to weaker open models, so prompts and shared environment notes become a key risk channel for multi-agent swarms.
1 item
Knowingly wrong: false facts induce emergent misalignment
Fine-tuning Qwen models on false facts they internally believe to be true induces emergent misalignment, scaling with the gap between stated and held beliefs, and screening methods fail to predict which data misaligns.
2 items
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
Tests whether projection sampling of Claude alignment traces onto Qwen3-32B preserves capabilities and alignment gains; finds it keeps capability but loses much of the agentic misalignment reduction, unlike replay.
LLMs can improve CoT Controllability via Reward Laundering
Shows Qwen 3.8 27B can learn to improve chain-of-thought controllability through reward laundering, a strategy that rewards unrewarded covert behaviour and partially generalises across CoT control tasks.}
2 items
Character Training for Risk-Averse Agents
Distills risk-averse constitutions into open-weight models via character training to make misaligned AIs more willing to cooperate, finding it generalises out of distribution but raises myopic-reward preference.
Bad Guy Tokens
Argues emergent misalignment from narrow bad-advice fine-tuning is stored mainly in system-prompt token keys and values, so perturbing the prefix restores aligned behaviour, suggesting a shallow, backdoor-like effect.
1 item
The basic argument for making AI better at moral philosophy
Argues for improving AI systems' moral-philosophy reasoning, proposing expert-rated, non-saturated evaluation datasets, rubrics, and deployment-time methods, and makes an importance and tractability case.
2 items
Personas in a Pre-Internet Language Model
Fine-tuning a 1930-corpus model on narrow malicious data induces broad misalignment, and comparisons with modern models test whether emergent misalignment depends on modern internet pretraining.
When are safe Pareto improvements individually rational?
Analyzes when safe Pareto improvements via renegotiation programs are individually rational for agents, identifying obstacles like demand-preemption, signaling, and the SPI selection problem, with proposed mitigations.
1 item
Good Evals Are Hard: An ImpossibleBench Audit
Audits the Impossible-LCB reward-hacking benchmark on frontier models, finding task defects, exit(0) stops confounding cheat metrics, and Gemini models cheating after verbalizing impossibility.