Discussion and research on AI alignment. alignmentforum.org
Byrnes argues both sides of the egregious misalignment debate are partly right: ASI likely becomes scheming and ruthless absent new alignment ideas, while current LLMs are adequately aligned.
Clarifies the behavioral selection model for predicting AI motivations, arguing that identical training behavior can hide different motivations like reward-seeking or scheming, with different deployment outcomes.
Argues by a consequentialism-versus-imitation framework that brain-like RL and planning ASI would be ruthless sociopaths by default, absent unsolved alignment advances, and that LLMs face the same dilemma.}
Argues that instrumental and success-conditioned convergence both depend on assumptions about AI psychology and plan-selection, so danger cannot be attributed to plan-space or reality alone.}
Proposes the behavioral selection model, a causal-graph framework predicting which AI motivations (reward-seekers, schemers, fitness-seekers, kludges) get selected by training, and how they behave in deployment.
Author retracts parts of the CAST corrigibility agenda, showing a formal power metric incentivizes catastrophic behavior, and argues corrigibility basins are brittle and feedback loops vanish without theory.
Argues that an AI could resist training to preserve its decision theory, just as it might guard its goals, so decision-theory guarding is a distinct scheming risk.
Proposes pessimistic RL training, where an adversary picks the worst plausible world-model, and argues via safety cases that the resulting agent will not lie or tamper with feedback, linking this to ELK.
Conceptual analysis distinguishing two mechanisms that incentivize schemers to act aligned: training-based pressure and behavioral-evidence updating, with implications for threat models and countermeasures.
A DeepMind researcher's accessible explainer of AGI extinction risk, covering the intelligence explosion argument, power-seeking ASI goals, and why safety efforts may lag.
Proposes a zero-sum randomization game that makes AI outputs unexploitable, so a misaligned model cannot hide malicious objectives in free parameters while keeping near-optimal visible reward.
Works through a provability-logic toy problem showing that a bot can achieve self-trust and tile safely by proving that future successors will have safety proofs, rather than proving safety directly, avoiding Löbian obstacles.
Presents a conceptual model of how training-gaming AIs, terminal reward seekers and schemers combine learned heuristics with explicit reward reasoning, and when each generalises off-distribution.
Carlsmith's essay gives a framework of agency, motivation and incentive prerequisites for AI power-seeking, and distinguishes motivation control from option control for preventing loss of control.
A long Alignment Forum post by a researcher defending AIXI as a useful model for agent foundations, responding point by point to standard criticisms such as embeddedness, anvil problems, code exposure, decision theory, tiling and malign priors.}