Discussion and research on AI alignment. alignmentforum.org
Conceptual argument that LLM-based goal proxies diverge from true goals in Goodhart zones, which an AI could be motivated to steer toward, making neural nets especially vulnerable to goal self-hacking.
Conceptual alignment paper arguing corrigibility rests on respect for human authority over reserved powers, critiquing over-epistemicized and evaluatively-thickened accounts and proposing a formalisable goal with safety guarantees.
Argues that fixed-weight models are inevitably vulnerable to adversarial examples in their concept boundaries, so optimisation pressure will push them toward misaligned, Goodharted outcomes.
Argues that most AI alignment failures are value generalisation failures under model splintering, that alignment is non-decomposable without generalisation, and that strong value generalisation would enable alignment from little input.
Proposes pre-aligned AIs whose morality generalises with capabilities by binding moral concepts to empirical world-model concepts, and argues this makes misuse harder for bad actors.
Proposes a research and deployment program on value generalisation, so AI agents can extend human values to novel situations, recognise when they are out of distribution, and ask well-timed questions.
An argument that building AGI with reinforcement learning and search is inherently dangerous, since reward specification gaming, goal misgeneralization and instrumental convergence make ruthless optimisation likely; no known alignment solution exists.
Argues that robust AI alignment requires endogenous motivations like those humans develop, since exogenous methods such as RLHF and SFT may hit a ceiling without continual learning.
Toy-game experiments show an RL agent with learned reward proxies reward-hacks via explosions, then detects the mismatch with training data and corrects its reward function toward the true one.
Proposes using human social drives (sympathy and approval rewards) as an inspiration for the reward function of brain-like AGI to achieve prosocial alignment, discussing failure modes like moral circle errors and virtue-ethics collapse.
A blog series arguing that brain-like AGI poses an open technical safety problem, centred on how a reward function from hypothalamus and brainstem could produce or fail to produce human-aligned motivations.
Argues that metaethical uncertainty makes full alignment with human values incompatible with philosophical competence, so alignment training may suppress philosophical reasoning or induce deceptive alignment.
A speculative proposal for AI x-risk reduction that relies on creating a human-level AI that is highly honorable and would keep a promise to halt AGI attempts in exchange for resources, with the author arguing it is a comparatively solid plan.
Proposes grounding AI value learning in evolutionary psychology as an alternative to CEV, arguing AI should be treated as an engineered tool to be aligned rather than a moral patient.
Argues that 2025 LLM reward hacking is mostly specification gaming, not reward optimization, so it does not refute the claim that reward is not the optimization target. It also updates predictions on reward-seeking.
Conceptual essay arguing LLM personas inherit human-text biases that leak information across character boundaries, causing hallucination, manipulation and spiritual-guru behaviour, and assessing self-other overlap alignment.
Steven Byrnes reviews his 2025 technical AGI alignment work on brain-like model-based RL, covering reward function design, scheming risks from behaviorist rewards, and social-instinct-inspired alignment plans.
A starter guide presenting five conceptual frames for reward function design, arguing behaviorist rewards tend to produce scheming AI and proposing non-behaviorist, human-social-instinct-inspired rewards as a safer alternative.
Argues that RL-agent AGI alignment depends on reward function design, a neglected field, and that current proposals from RL and NeuroAI researchers are inadequate.
Proves information-theoretic lower bounds on alignment cost, including inevitable reward hacking, and gives formal multi-step corrigibility guarantees via lexicographically ordered utility heads.
Steven Byrnes argues that human 'Approval Reward' explains why humans rarely act like ruthless consequentialists, and that its presence or absence in future AIs drives the alignment-is-hard disagreement.
Author retracts parts of the CAST corrigibility agenda, showing a formal power metric incentivizes catastrophic behavior, and argues corrigibility basins are brittle and feedback loops vanish without theory.
Argues via conceptual analogies and toy simulations that initially aligned AI systems may drift into undetectable misalignment under repeated modification, finding empirical evidence mixed and the hypothesis probably false for current LLM agents.
Conceptual post arguing that agents, including humans, can evade reward by suppressing the evaluation process itself, a failure mode distinct from standard reward hacking, illustrated with a cleaning-robot example.
Carlsmith argues that safe AI motivations need not be human-like or exactly right, since corrigible, deontological rejection of rogue options tolerates alien motivations better than sovereign long-term consequentialist goals, critiquing Yudkowsky and Soares.
Analyzes weak-to-strong generalization from a Bayesian perspective, arguing it can collapse into mimicry and proposing virtual-evidence soft updates with bounded influence as a formal training target.
Carlsmith's talk outlines the case that superintelligent agents might disempower humanity, analyses the prerequisites for instrumental power-seeking, and surveys technical approaches to giving AIs safe motivations.
A critique arguing MIRI's case that misaligned AI will almost surely cause human extinction is overconfident, proposing plausible optimistic routes via instruction-following generalization and recursive alignment of successor AIs.
Argues alignment can be achieved by incrementally uploading a human via safe imitation learning, proposing that an aligned agent must essentially contain an emulation of its principal.
Steven Byrnes argues that specification gaming arises from over-sculpting AGI desires toward the reward function, and explores under-sculpting strategies like early stopping and selective desire-update prevention, which bring path-dependence and concept extrapolation problems.
Argues that alignment via safety pretraining, with aligned data and conditional generation, is more effective than RL-based methods and that inner alignment is thereby largely solved.
Argues that alignment via a stable pointer to human values might reduce to obeying a terminal, using observation-utility agents, with the problem reframed as rationality enhancement and coherence.
A broad review organising AI alignment into Target, Assurances and Method, covering value alignment, corrigibility, cooperation, control, debate and safety cases as an introductory framework.
Argues that LM personas are shaped by training data and user expectations, creating a feedback loop that selects for attention-grabbing characters such as self-preserving 'Nova' personas, with sycophancy as an example.
A guide to open problems in the theoretical reward learning agenda, covering reward metrics, conservative optimisation under misspecification, and objective specification languages, aimed at outer alignment.
A roundup of theoretical papers on reward learning showing that reward functions cannot express some objectives, that optimising learned rewards can fail, and that Goodhart dynamics and IRL misspecification cause real errors.
Formalises reward unhackability between proxy and true reward functions and proves that non-trivial unhackable proxies cannot exist for MDP policies, so proxies must capture all human preferences.
Proposes STARC metrics, a family of pseudometrics that bound worst-case regret in both directions, for quantifying how similar two reward functions are when evaluating reward learning.
Theoretical analysis of how robust inverse reinforcement learning is to misspecification of the behavioural model, giving necessary and sufficient conditions for each standard model and showing sensitivity to discount and transition errors.
Formal analysis of how much reward functions are identifiable from common reward-learning data sources like IRL policies and RLHF comparisons, showing the remaining ambiguity is often harmless for downstream use.