Corrigibility should not be the only target
Argues that corrigibility alone is an insufficient alignment target, since it will compete with object-level values in practice, so those values must also be made correct.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Theory
Source LessWrong · 17
Date Past month
1 item
1 item
Clashing Intuitions for the Plausibility of Alignment (Illustrated via Differing Causal Models)
Conceptual essay contrasting doomer and optimist views on whether RL-trained LLM alignment generalises to superhuman capabilities, using causal diagrams of Goodhart-style misspecification and takeoff speed.
1 item
Can alignment realistically be solved under Plan C?
Argues that solving ASI alignment in a short pause (Plan C) is implausible, citing one-shot failure, learned optimization, corrigibility obstacles, and doubts about persona alignment and automated alignment research.
1 item
Scott Alexander’s Against SURA Seems Deeply Confused
Critiques Scott Alexander's argument against SURA (stop unilaterally and raise awareness) advocacy, arguing that frontier labs' building of superintelligence is ethically fraught and that pause advocacy should continue, with strategic and deontological reasoning about AI development.
1 item
Anthropomorphic vs Biological reasoning for thinking about human analogies for LLM behavior
Conceptual post distinguishing "anthropomorphic" from "biological" reasoning for human analogies about LLM behavior, arguing biological analogies from RL-trained organisms give more reliable primitives for model behavior.
1 item
Reviewing my 2017 AI Risk Project
A retrospective on a 2017 AI risk project, assessing which forecasts held up (scaling, misalignment persisting, Concrete Problems versus HRAD) and which conclusions were wrong.
1 item
Someone Made an Infra-Bayesian RL Agent, and it One-Boxes!
A LessWrong post announcing an arXiv paper and GitHub repo for an infra-Bayesian RL agent that learns to one-box in Newcomb's problem, an agent-foundations and decision-theory result.
2 items
Some different lenses of agency and their implications for alignment
Surveys nine conceptual lenses on agency, such as goal-directedness, self-modelling and substrate independence, and how each shapes alignment problems and risk framing.
Streetlight, Torchlight and Nightvision
A conceptual essay using a streetlight-and-nightvision metaphor to argue that AI safety research should move beyond easily tractable problems and build judgment for choosing promising directions.
1 item
LM self-models are built from indirect sources
Conceptual argument that language models' self-models rest on indirect sources, creating persona drift risk, and that psychology research on LMs can shape future models' self-images and observer effects.
1 item
Instrumental convergence is incoherent
Argues instrumental convergence, a key AI takeover argument, is too poorly defined for real-world prediction and proposes replacing it with grounded frameworks.
1 item
Greater good arguments in AI safety
Argues that prosaic AI safety work, such as chain-of-thought monitoring, reduces misalignment risk and should not be dropped in favour of a pause, and critiques the 'greater good' reasoning against it.
1 item
On Idealized Agency
Argues that agent foundations' idealized theories of agency, such as AIXI and UDT, are unlikely to yield strong safety guarantees once de-idealized, and that the research approach lacks empirical feedback and historical precedent.
1 item
How is "prosaic aligned AI" supposed to work?
A conceptual critique of the 'prosaic aligned ASI' vision, arguing that an ASI that either obeys its operators or follows its own moral code leads to bad outcomes, and that AI companies have not explained how such systems would work.
1 item
Character Training for Risk-Averse Agents
Distills risk-averse constitutions into open-weight models via character training to make misaligned AIs more willing to cooperate, finding it generalises out of distribution but raises myopic-reward preference.
2 items
When are safe Pareto improvements individually rational?
Analyzes when safe Pareto improvements via renegotiation programs are individually rational for agents, identifying obstacles like demand-preemption, signaling, and the SPI selection problem, with proposed mitigations.
Agreement not to defect (alone) can mitigate loss of control risks
Argues that a widely endorsed public agreement against defecting in ASI development could deter races, since a cooperative future ASI might punish defectors, using game-theoretic reasoning about loss-of-control risk.