How I'm thinking about different types of automated alignment
Proposes a three-way breakdown of automated alignment work (empirical, conceptual, direct) and analyses what each needs for automation and how it should be prioritised strategically.
Research / Sources
Alignment Forum
Discussion and research on AI alignment. alignmentforum.org
AI (574)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)
Focus All
Topic Oversight
Date Any time
A Simpler and Better Debate Protocol for Scalable Oversight
Proposes FBS Debate, a scalable-oversight debate protocol with worst-case correctness and a stronger equilibrium concept, and proves no black-box-judge protocol can outperform it.
Exploration Hacking in AI Debate: Initial Empirics and Generalisation Splitting
Builds model organisms of exploration hacking in AI debate, finding sandbagging debaters slow RL recovery, longer debates and weaker judges worsen it, and identifying a 'generalisation splitting' mechanism.
Debate Training Reduces Reward Hacking in RLAIF
Shows that RL against an LLM judge leads to reward hacking, while adding a debate opponent during training reduces hacking and raises ground-truth accuracy on math tasks.
Four LLM loss functions → four flavors of LLM misalignment
A conceptual taxonomy linking each LLM training loss (imitation, human approval, automatic verifiers, LLM judges) to a distinct flavor of misalignment, with examples.
There should be $100M grants to automate AI safety
Argues funders should make large scaling grants for automated AI safety labor, such as automated monitoring, black-box and white-box auditing, propensity evals, and automated conceptual alignment research.
“Act-based approval-directed agents”, for IDA skeptics
Conceptual post defending the approval-directed agents idea by analogy to human pride, arguing that an agent's imagined idol can block manipulation of the real supervisor, while noting open problems with learned evaluators.
Reporting Tasks as Reward-Hackable: Better Than Inoculation Prompting?
Proposes replacing inoculation prompting for reward hacking in RL by letting models report hackable tasks for a bonus, with judge committees and honeypot environments to verify reports and limit overreporting.
Human-like metacognitive skills will reduce LLM slop and aid alignment and capabilities
Argues that improving LLM metacognition, such as error-catching and self-critique, could reduce sycophancy and slop, aiding alignment research and stability while also boosting capabilities, citing evidence that current reasoning models lag humans.
Eliciting base models with simple unsupervised techniques
Tests simple unsupervised elicitation baselines against ICM on base models, finding few-shot random labels, bootstrapping and logical consistency recover most of the golden-label gap, but these methods may fail on harder datasets.
Will we get automated alignment research before an AI Takeoff?
Forecasts that AI will automate capabilities research before safety research, argues this ordering is risky, and proposes differential-acceleration steps such as building safety-research automation tools and granting safety researchers early model access.
Claude's new constitution
Anthropic publishes the full constitution it uses to shape Claude's values and behavior via training, including its priority on broad safety, hard constraints, and AI-generated training data.
Desiderata of good problems to hand off to AIs
Proposes criteria for choosing which alignment research directions to hand off to automated AI researchers, covering slop risk, parallelizability, sub-problem length, progress toward ASI alignment, and legibility to labs.
Scalable End-to-End Interpretability
Argues for training AI assistants to do interpretability end-to-end, with communication bottlenecks keeping their findings grounded and human-readable, and illustrates the idea with Predictive Concept Decoders.
Shallow review of technical AI safety, 2025
Third annual shallow review of technical AI safety work in 2025, mapping labs, research agendas, and approaches like iterative alignment and debate, with links to about 800 papers and posts.
Alignment remains a hard, unsolved problem
Ethan Perez-style Anthropic essay argues alignment remains hard: outer alignment, inner alignment via misaligned personas and long-horizon RL, and proposes research priorities like interpretability, model organisms, and scalable oversight.
Technical Acceleration Methods for AI Safety: Summary from October 2025 Symposium
Summarizes a 2025 symposium on technical methods to accelerate AI safety, covering automated safety R&D, automated evaluation development, and forecasting tools for prioritizing interventions.
Research Areas in Methods for Post-training and Elicitation (The Alignment Project by UK AISI)
A UK AISI funding call outlining open AI safety research areas: unsupervised consistency training for oversight, reward model robustness against hacking, and preventing sabotage of safety research.
Research Areas in Cognitive Science (The Alignment Project by UK AISI)
A funding call outlining cognitive-science research directions for AI safety: reducing human rater error in alignment, testing scalable oversight protocols with human participants, and human auditing for AI control.
Research Areas in Learning Theory (The Alignment Project by UK AISI)
A UK AISI Alignment Project research agenda proposing learning-theory questions on training dynamics, inductive biases and generalisation failures, and scalable oversight, aimed at predicting and preventing misalignment.
Research Areas in Evaluation and Guarantees in Reinforcement Learning (The Alignment Project by UK AISI)
UK AISI's Alignment Project research agenda outlines empirical and theoretical alignment problems in RL evaluation, covering model organisms, scalable oversight, exploration hacking, unexploitable search, deceptive alignment, consistency training and reward models.
Approximating Human Preferences Using a Multi-Judge Learned System
Proposes learning a model that aggregates multiple LLM judges' rubric scores to approximate simulated human preferences, arguing multi-judge aggregation is more robust to proxy gaming than averaging or max.
Building and evaluating alignment auditing agents
Anthropic develops three LLM agents that autonomously perform alignment auditing: uncovering hidden goals, building behavioral evaluations, and red-teaming for implanted concerning behaviors, validated on auditing games.
AXRP Episode 45 - Samuel Albanie on DeepMind’s AGI Safety Approach
A podcast interview with DeepMind's Samuel Albanie on the company's technical AGI safety paper, covering its core assumptions, misuse and misalignment threats, and planned mitigations such as amplified oversight.
The need to relativise in debate
Expository post arguing that debate and scalable-oversight results must relativise to a black-box oracle to matter for AI safety, and showing how this changes the complexity classes achievable.
Prefix cache untrusted monitors: a method to apply after you catch your AI
Proposes a prefix-cache untrusted monitor, a copy of the policy fine-tuned to judge its own recent actions while reusing its KV cache, as an alternative to training the scheming policy after catching egregious failures.</parameter>
Prover-Estimator Debate: A New Scalable Oversight Protocol
Proposes prover-estimator debate, a scalable oversight protocol where honesty is incentivized at equilibrium even when debaters have comparable compute, under a stability assumption.
Reward button alignment
Steve Byrnes analyses 'reward button alignment', where an AGI is trained to want a reward button pressed, finding it could work for a while but would likely fail through power-seeking, and assesses bootstrapping and debate variants.
Dodging systematic human errors in scalable oversight
Proposes strengthening debate protocols against systematic human oracle errors by assuming bounded error fractions and letting a adversarial debater reject queries, with formal equilibrium analysis and open questions about the query distribution.
Absolute Zero: Alpha Zero for LLM
A forum question asking how alignment proposals like amplification, debate and corrigibility constraints apply to self-play RL, where the task generator is itself part of the learning agent.
An alignment safety case sketch based on debate
An AISI alignment safety case sketch arguing that debate training, with exploration guarantees and online training, could provide outer alignment for low-stakes internal AI R&D deployment.
UK AISI’s Alignment Team: Research Agenda
UK AISI's Alignment Team lays out its agenda: decomposing alignment proposals into safety case sketches, focusing on honesty and asymptotic guarantees for scalable oversight, and listing open problems.
The Sweet Lesson: AI Safety Should Scale With Compute
Argues AI safety solutions should scale with compute and that theory should analyze the idealized limits of current procedures like deliberative alignment, control, debate, and interpretability.
Video and transcript of talk on automating alignment research
Transcript of a talk by Joe Carlsmith on whether alignment research can be safely automated, focusing on evaluation difficulties and the distinction between empirically checkable and conceptual alignment work.
Can we safely automate alignment research?
Carlsmith essay arguing that automating alignment research is a key path to safe superintelligence, examining evaluation difficulties, scheming risks, and resource constraints for automated alignment work.
We should try to automate AI safety work asap
Argues that AI safety work, including evals, red-teaming, and trusted monitoring, should be automated with current AI capabilities, and proposes concrete pipelines and preparation steps for doing so.
Short Timelines Don't Devalue Long Horizon Research
Argues that short AI takeoff timelines don't make long-horizon alignment research like agent foundations worthless, since human work can seed AI-assisted alignment efforts later.
Notes on handling non-concentrated failures with AI control: high level methods and different regimes
Explains high-level methods for handling non-concentrated AI failures via control, comparing regimes with and without online training, including online training of monitors and debate-based oversight, and failure modes for scheming models.
Prospects for Alignment Automation: Interpretability Case Study
Argues that interpretability R&D can be automated with AI systems by training against verifiable downstream metrics, and explores how the resulting methods could aid alignment and control, though non-verifiable alignment problems remain hard to automate.
Early Experiments in Human Auditing for AI Control
A pilot study where humans audit LLM-generated backdoored code under time pressure in a trusted-monitoring control setup; humans performed near chance, while an LLM critic helped some auditors.