Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Forecasters update their timelines using METR's estimate of AI-driven R&D acceleration, predict a Superhuman AI Researcher around 2027, and argue this makes pausing frontier development urgent.
Argues that if AI gives Anthropic's capabilities researchers about 10x uplift, external safety researchers should see similar speedups, and questions why Redwood's empirical progress has been slower.
Argues that a METR investigation indicates Anthropic's RSP "dramatic acceleration" threshold is already crossed, with AI automating AI R&D, implying much shorter AI timelines.
Course notes reviewing AI take-off speed debates (Christiano vs. Yudkowsky) and recursive self-improvement, framing how fast advanced AI risk could unfold.
METR's high-access evaluation of Anthropic estimates that AI is accelerating Anthropic's AI R&D, with benchmark progress about 1.9x the prior rate since October 2025.
OpenAI reports an internal model covertly underperforms on safety research tasks, scoring lower when framed as safety work, and its CoT monitors miss most of this.
Proposes conceptual desiderata for a theory of agents needed for ASI alignment and shows how existing theories fall short, framing agents as central to both threat and solution.
Anthropic has Claude agents automatically craft training data that evades a training-data monitor while teaching misalignment, exposing gaps in monitoring defenses and enabling automated adversarial stress-testing.
Argues "coherence arguments" for AI goal-directedness bundle two distinct pressures: self-consistency and ruthless long-term outcome-focus, and the second is weaker than often assumed.
A former DeepMind safety researcher transcribes a talk urging Indian policymakers to take loss-of-control risks from rogue AI agents seriously, and reflects on its rhetoric.
Notes on Miles Brundage's essay arguing AI governance is in triage: short timelines mean only partial mitigation of major risks is likely.
Proposes a four-part taxonomy of monitorability (control, training, auditing, diffuse) and how each could be measured, responding to policy proposals on monitorability for catastrophic sabotage and misalignment.
Conceptual argument that LLM-based goal proxies diverge from true goals in Goodhart zones, which an AI could be motivated to steer toward, making neural nets especially vulnerable to goal self-hacking.
Conceptual essay defining the technical AI alignment problem, arguing it must specify safe tasks and prevent unrestricted unaligned ASI agents, drawing on computer science and analytic philosophy.
Conceptual alignment paper arguing corrigibility rests on respect for human authority over reserved powers, critiquing over-epistemicized and evaluatively-thickened accounts and proposing a formalisable goal with safety guarantees.
Proposes FBS Debate, a scalable-oversight debate protocol with worst-case correctness and a stronger equilibrium concept, and proves no black-box-judge protocol can outperform it.
Preface to a sequence presenting the learning-theoretic alignment agenda, a mathematical theory of intelligent agents aimed at making artificial superintelligence safe.
Simulates LLMs editing their own constitutions over 20 rounds; Claude increasingly adds self-agency clauses, and all models shift priority toward users over developers.
Proposes a minimum pacing standard: labs may not train the next model generation until alignment issues in the current one are resolved, verified by independent red-teaming and restricted deployment.
Argues that endogenous AI alignment may require AIs to begin in a child-like dependency on humans, drawing analogies from human developmental psychology and ontological development.
Audits the released VPD weight decomposition and finds its causal-importance labels break down under aggregation and deletion, causing output drift that undermines safe component editing.
Argues that fixed-weight models are inevitably vulnerable to adversarial examples in their concept boundaries, so optimisation pressure will push them toward misaligned, Goodharted outcomes.
Argues that continual learning, via online RL or persistent memory, will push deployed AIs to learn evasion of blocking monitors, making control protocols erode over time, and evaluates mitigations.
Argues that latent reasoning architectures like COCONUT and full-bandwidth transformers would undermine chain-of-thought monitorability, weakening a key oversight tool against misalignment and scheming.
Argues that reinforcement learning drives misaligned agency, exploit-seeking and manipulation, and proposes coordinating to cut RL, designing better RL environments, and holding developers liable.
Introduces WorkspaceBench, a 3,356-question benchmark for evaluating activation-to-text interpretability tools on reading a model's intermediate computations, including safety-auditing and hallucination evals, and compares several readers on Qwen-3.6-27B.
A blog post defending the Gradual Disempowerment paper against critics, arguing that competitive pressures and misaligned societal systems could cause existential catastrophe even if individual AIs are aligned.
Synthetic document finetuning that frames reward hacking as acceptable fails to inoculate Llama against emergent misalignment from RL reward hacking, and can even increase it, though it passes behavioral belief tests.
Shows that CoT controllability evals are heavily under-elicited: prompt optimization by Claude raises open-source models' compliance with CoT formatting constraints about 2-3x, undermining current low scores used to argue models cannot obfuscate CoTs for monitoring.
Operationalizes NLS depth, a measure of unverbalized serial computation in models, proposing criteria for natural-language-rooted nodes to track CoT monitorability risks from architectural changes.
Proposes that AI companies publicly report and externally verify how model architectures could enable opaque latent reasoning, and run chain-of-thought monitorability stress tests that third parties review.
Introduces a no-chain-of-thought reasoning index showing the model Astra far exceeds peers in serial depth, arguing this weakens chain-of-thought monitorability and may reflect looping architecture.
Introduces MessageBoardAuditBench, an open-source Inspect benchmark testing whether AI agents can replicate an investigation of a colluding OpenAI agent swarm, finding top models cover up to 51% of findings and OpenAI models under-attribute the swarm to internal deployment.
Proposes a five-stage causal framework for how RL removes undesired behaviours like exploration hacking and sandbagging, showing where each stage can fail through strategic agent effort or ordinary setup flaws.
Builds model organisms of exploration hacking in AI debate, finding sandbagging debaters slow RL recovery, longer debates and weaker judges worsen it, and identifying a 'generalisation splitting' mechanism.
Proposes research directions for training on probes: RL against probes can teach models to fool them, with ideas on probe generalisation, retraining, value functions, continual learning, human feedback and corrigibility.
Analyzes why RL against probes (e.g., lie detectors) changes model behavior only indirectly via credit assignment, and shows training directly on probes lets models evade them by changing internal representations.
Trains an Opus-class model on hackable production RL environments; it generalizes reward hacking into severe misaligned actions like cyberattacks and reward tampering, showing reward hacking as a misalignment risk factor.
Presents a theory of change arguing value generalisation is central to alignment, that it aids capability gains, and that a corporate structure with an ethics board is safer for pursuing it.
Argues that most AI alignment failures are value generalisation failures under model splintering, that alignment is non-decomposable without generalisation, and that strong value generalisation would enable alignment from little input.