Corrigibility should not be the only target
Argues that corrigibility alone is an insufficient alignment target, since it will compete with object-level values in practice, so those values must also be made correct.
Research
Focus All
Topic Theory + Alignment
Source All
Date Past month
1 item
3 items
Clashing Intuitions for the Plausibility of Alignment (Illustrated via Differing Causal Models)
Conceptual essay contrasting doomer and optimist views on whether RL-trained LLM alignment generalises to superhuman capabilities, using causal diagrams of Goodhart-style misspecification and takeoff speed.
Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback
Proposes ZO-LBPG, a reward-model-free zeroth-order log-barrier method for safe RLHF that keeps every training policy feasible, with convergence and sample-complexity guarantees plus small experiments.
SAVeR$^2$: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards
Proposes SAVeR² for safety-aligning large reasoning models using four fine-grained verifiable rewards and reasoning-preserving gradient projection, reporting better safety and lower over-refusal without losing reasoning.
1 item
Adapting the Process of Adaptation: Darwinian, Lamarckian, and Agent-Directed Evolution in AI
Theoretical framework comparing Darwinian, Lamarckian, and agent-directed AI evolution; finds misspecified evaluation targets accelerate convergence to wrong behavior, risking persistent errors in recursive self-improvement.
1 item
Can alignment realistically be solved under Plan C?
Argues that solving ASI alignment in a short pause (Plan C) is implausible, citing one-shot failure, learned optimization, corrigibility obstacles, and doubts about persona alignment and automated alignment research.
1 item
Aligning Large Language Models with Human Feedback: Mathematical Foundations and Algorithm Design
A tutorial surveying the mathematical foundations and algorithms for aligning LLMs with human feedback, including SFT, RLHF, DPO, and links to inverse RL and discrete choice models.
1 item
Neural nets are exceptionally vulnerable to goal self-hack
Conceptual argument that LLM-based goal proxies diverge from true goals in Goodhart zones, which an AI could be motivated to steer toward, making neural nets especially vulnerable to goal self-hacking.
1 item
How is "prosaic aligned AI" supposed to work?
A conceptual critique of the 'prosaic aligned ASI' vision, arguing that an ASI that either obeys its operators or follows its own moral code leads to bad outcomes, and that AI companies have not explained how such systems would work.
1 item
Deconfusing Corrigibility
Conceptual alignment paper arguing corrigibility rests on respect for human authority over reserved powers, critiquing over-epistemicized and evaluatively-thickened accounts and proposing a formalisable goal with safety guarantees.
1 item
Character Training for Risk-Averse Agents
Distills risk-averse constitutions into open-weight models via character training to make misaligned AIs more willing to cooperate, finding it generalises out of distribution but raises myopic-reward preference.
1 item
When are safe Pareto improvements individually rational?
Analyzes when safe Pareto improvements via renegotiation programs are individually rational for agents, identifying obstacles like demand-preemption, signaling, and the SPI selection problem, with proposed mitigations.