Corrigibility should not be the only target
Argues that corrigibility alone is an insufficient alignment target, since it will compete with object-level values in practice, so those values must also be made correct.
Research
Focus All
Topic Theory + Alignment
Source All
Date Any time
1 item
3 items
Clashing Intuitions for the Plausibility of Alignment (Illustrated via Differing Causal Models)
Conceptual essay contrasting doomer and optimist views on whether RL-trained LLM alignment generalises to superhuman capabilities, using causal diagrams of Goodhart-style misspecification and takeoff speed.
Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback
Proposes ZO-LBPG, a reward-model-free zeroth-order log-barrier method for safe RLHF that keeps every training policy feasible, with convergence and sample-complexity guarantees plus small experiments.
SAVeR$^2$: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards
Proposes SAVeR² for safety-aligning large reasoning models using four fine-grained verifiable rewards and reasoning-preserving gradient projection, reporting better safety and lower over-refusal without losing reasoning.
1 item
Adapting the Process of Adaptation: Darwinian, Lamarckian, and Agent-Directed Evolution in AI
Theoretical framework comparing Darwinian, Lamarckian, and agent-directed AI evolution; finds misspecified evaluation targets accelerate convergence to wrong behavior, risking persistent errors in recursive self-improvement.
1 item
Can alignment realistically be solved under Plan C?
Argues that solving ASI alignment in a short pause (Plan C) is implausible, citing one-shot failure, learned optimization, corrigibility obstacles, and doubts about persona alignment and automated alignment research.
1 item
Aligning Large Language Models with Human Feedback: Mathematical Foundations and Algorithm Design
A tutorial surveying the mathematical foundations and algorithms for aligning LLMs with human feedback, including SFT, RLHF, DPO, and links to inverse RL and discrete choice models.
1 item
Neural nets are exceptionally vulnerable to goal self-hack
Conceptual argument that LLM-based goal proxies diverge from true goals in Goodhart zones, which an AI could be motivated to steer toward, making neural nets especially vulnerable to goal self-hacking.
1 item
How is "prosaic aligned AI" supposed to work?
A conceptual critique of the 'prosaic aligned ASI' vision, arguing that an ASI that either obeys its operators or follows its own moral code leads to bad outcomes, and that AI companies have not explained how such systems would work.
1 item
Deconfusing Corrigibility
Conceptual alignment paper arguing corrigibility rests on respect for human authority over reserved powers, critiquing over-epistemicized and evaluatively-thickened accounts and proposing a formalisable goal with safety guarantees.
1 item
Character Training for Risk-Averse Agents
Distills risk-averse constitutions into open-weight models via character training to make misaligned AIs more willing to cooperate, finding it generalises out of distribution but raises myopic-reward preference.
1 item
When are safe Pareto improvements individually rational?
Analyzes when safe Pareto improvements via renegotiation programs are individually rational for agents, identifying obstacles like demand-preemption, signaling, and the SPI selection problem, with proposed mitigations.
1 item
Culture in the Code: Anthropological Concepts Decoding AI's Hidden Assumptions
Uses anthropological concepts to critique alignment's assumptions that values are behaviorally inferable, quantifiable and optimizable, and proposes 'thick alignment' grounded in social practice and community interpretation.
1 item
The Asymptotics of Language Model Alignment with Memory
Extends the asymptotic equivalence of best-of-n and KL-constrained RL alignment from i.i.d. to Markovian token sequences, and characterises when their output distributions match exactly for short outputs.
1 item
Revisiting scaling laws for reward optimization
Derives and empirically validates a scaling law for reward over-optimization in alignment, showing performance depends on the square root of min(log comparisons, KL budget), with an information-theoretic model.
2 items
Fixed-weight models are adversarially vulnerable: hence misaligned
Argues that fixed-weight models are inevitably vulnerable to adversarial examples in their concept boundaries, so optimisation pressure will push them toward misaligned, Goodharted outcomes.
Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control
Analyses how imperfect verifiers in RLVR let reward rise while correctness falls, shows observational limits on detecting accepted errors, and proposes audit-based feedback that selectively reduces them.
1 item
Axiom Satisfiability of Linear Rewards in Alignment
Shows that enforcing social-choice axioms like Pareto optimality on linear preference rewards costs bounded slack, and proposes a method trading slack against violations that beats linear BTL in experiments.
1 item
Thoughts on the persona selection model
Analyzes the persona selection model of LLM behavior, arguing it is over-applied, has limited predictive power, and that persona conditionalization is an important but overlooked wrinkle for AI risk reasoning.
1 item
Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
Develops a comparative framework and formal bounds showing how reward hacking of evaluator proxies arises across weight updates, output selection, and persistent prompts, and how defenses transfer.
1 item
Mitigating Reward Hacking as Institutional Design
Argues reward hacking stems from reward misspecification and information bottlenecks in RL, and proposes reframing it as institutional design, including agent appeal mechanisms and uncertainty-aware objectives.
1 item
Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner
Gives tight bounds on the utility distortion of RLHF under heterogeneous preferences, showing the exponential blow-up comes from mismatch between the preference-data distribution and the reference policy rather than from RLHF itself.
1 item
Attunement, not alignment
Argues that aligning AI to human moral judgments is misguided and proposes training agents to attune to societal flourishing and plurality, as AlphaZero learned chess without human games.
1 item
Variance of Value
Argues that training an AI with RL on a reward equal to human utility fails unless it generalizes from low-stakes to high-stakes value, which current methods are unlikely to do.
3 items
[Macroagents] 2. Design lenses for optimizing macroagents
A conceptual framework post on designing macroagents (organizations and AI-agent collectives) for alignment, epistemics and efficiency, covering incentive failures, selection effects, and RL incentives for AI.
Value generalisation Theory of Change: the theory behind the approach
Argues that most AI alignment failures are value generalisation failures under model splintering, that alignment is non-decomposable without generalisation, and that strong value generalisation would enable alignment from little input.
AI Alignment through a Game-theoretic Lens: A Survey
A survey reviewing LLM and AI-agent alignment through a game-theoretic lens, organising work around preference diversity, alignment priority, and temporal dynamics, and assessing where game theory helps.
1 item
A Survey on Rubric-Guided Reinforcement Learning for Language Models
A survey of rubric-guided RL for aligning LLMs, offering a Bayesian framing of constitutions and rubrics, a taxonomy of methods, and an analysis of linguistic reward hacking and semantic drift.
1 item
Algorithmic Impact Reveals the Hidden Social Choice Structure of Alignment
Reframes AI alignment as social choice over a convex welfare-impact space, showing voting and random-dictatorship protocols are strategyproof and unanimous, and deriving welfare-maximising protocols under harm constraints.
1 item
Some reasons alignment doesn’t generalise well
Argues that capabilities generalise out of distribution through self-correction and simplicity bias, while alignment does not, because values have no external ground truth and training selects for wrong proxies.
2 items
Kimi likes causal decision theory more after RL in twin prisoner’s dilemmas
Shows that RL on twin prisoner's dilemmas makes Kimi K2.6 more sympathetic to causal decision theory, including on abstract DTBench questions, raising concerns about multi-agent training effects on model decision-theoretic propensities.
Towards a theory of inference-time alignment with unknown rewards
Formulates inference-time alignment as a PAC-style weak-to-strong learning problem, introduces an alignment dimension that characterizes when reward classes are alignment learnable, and gives a tournament-based learning procedure.
1 item
Toward a Theory of Value in AI Alignment
Annotates 94 AI value alignment papers and finds most leave human values undefined, relying on preferences that risk reducing culturally situated values to binary choices, especially as synthetic data and autoraters replace human annotators.
1 item
Metanormative Theory for RL-Based Moral Agents
A philosophical analysis that applies metanormative theory to RL-based machine ethics and value alignment, offering criteria for when an RL agent's behavior counts as moral and for comparing approaches.
1 item
The Off-Support Barrier: Why Semantic Safety Constraints Are Not Learning-Problem Invariants, and What Follows for Prior Design, Containment, and Verification
Argues that semantic safety constraints are off-support objects that singular learning theory's RLCT cannot measure, deriving consequences for reward hacking, prior design, harness-versus-model placement, and formal verification.
1 item
Taboo “equilibrium”: Less confused frames for research on AI bargaining
Argues that AI bargaining research should use mechanistic explanations and constraints on beliefs rather than Nash equilibrium heuristics, using miscoordination in safe Pareto improvement settings as the running example.
1 item
Fragility of Value under Imperfect Alignment
A formal model of alignment training shows that agents with catastrophic value functions can still be deployed when proxy conditions are imperfect, motivating limits on optimization pressure such as quantilizers.
Also: LessWrong blog
2 items
Value Generalisation 3: Pre-aligned AIs
Proposes pre-aligned AIs whose morality generalises with capabilities by binding moral concepts to empirical world-model concepts, and argues this makes misuse harder for bad actors.
Value Generalisation 1: a Research and Deployment Program
Proposes a research and deployment program on value generalisation, so AI agents can extend human values to novel situations, recognise when they are out of distribution, and ask well-timed questions.
1 item
RL & search is a terrifying way to build AGI (an FAQ)
An argument that building AGI with reinforcement learning and search is inherently dangerous, since reward specification gaming, goal misgeneralization and instrumental convergence make ruthless optimisation likely; no known alignment solution exists.