How Hard a Problem is Alignment? (My Opinionated Answer)
An opinionated essay estimating how hard AI alignment is by comparing person-years of effort against steam engine safety, Apollo, and P vs NP, and arguing for accelerating the field's growth.
Research / Sources
Alignment Forum
Discussion and research on AI alignment. alignmentforum.org
AI (574)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)
Focus All
Topic All
Date Any time
Human-like metacognitive skills will reduce LLM slop and aid alignment and capabilities
Argues that improving LLM metacognition, such as error-catching and self-critique, could reduce sycophancy and slop, aiding alignment research and stability while also boosting capabilities, citing evidence that current reasoning models lag humans.
How do we (more) safely defer to AIs?
Discusses how to safely defer to increasingly capable AIs for AI safety and strategy work, covering the basin of good deference, required alignment and capability profiles, and the lead time needed to manage exogenous risks.
Will we get automated alignment research before an AI Takeoff?
Forecasts that AI will automate capabilities research before safety research, argues this ordering is risky, and proposes differential-acceleration steps such as building safety-research automation tools and granting safety researchers early model access.
A Conflict Between AI Alignment and Philosophical Competence
Argues that metaethical uncertainty makes full alignment with human values incompatible with philosophical competence, so alignment training may suppress philosophical reasoning or induce deceptive alignment.
Grounding Value Learning in Evolutionary Psychology: an Alternative Proposal to CEV
Proposes grounding AI value learning in evolutionary psychology as an alternative to CEV, arguing AI should be treated as an engineered tool to be aligned rather than a moral patient.
Technical Acceleration Methods for AI Safety: Summary from October 2025 Symposium
Summarizes a 2025 symposium on technical methods to accelerate AI safety, covering automated safety R&D, automated evaluation development, and forecasting tools for prioritizing interventions.
The Best Way to Align an LLM: Is Inner Alignment Now a Solved Problem?
Argues that alignment via safety pretraining, with aligned data and conditional generation, is more effective than RL-based methods and that inner alignment is thereby largely solved.
Video and transcript of talk on automating alignment research
Transcript of a talk by Joe Carlsmith on whether alignment research can be safely automated, focusing on evaluation difficulties and the distinction between empirically checkable and conceptual alignment work.
Can we safely automate alignment research?
Carlsmith essay arguing that automating alignment research is a key path to safe superintelligence, examining evaluation difficulties, scheming risks, and resource constraints for automated alignment work.
We should try to automate AI safety work asap
Argues that AI safety work, including evals, red-teaming, and trusted monitoring, should be automated with current AI capabilities, and proposes concrete pipelines and preparation steps for doing so.
Prospects for Alignment Automation: Interpretability Case Study
Argues that interpretability R&D can be automated with AI systems by training against verifiable downstream metrics, and explores how the resulting methods could aid alignment and control, though non-verifiable alignment problems remain hard to automate.
AI for AI safety
Carlsmith argues that using frontier AI labor to strengthen safety progress, risk evaluation, and capability restraint, especially automated alignment research, is crucial for avoiding loss of control from an AI capability feedback loop.
Deep sparse autoencoders yield interpretable features too
Proposes deep sparse autoencoders with non-sparse layers around the sparse layer on GPT-2-small activations. Finds their features are at least as interpretable as shallow SAE features, though dead neurons remain a problem.
How might we safely pass the buck to AI?
Argues AI safety researchers should aim to safely hand off alignment research to AI agents, outlining strategies and four arguments justifying trust in those agents, including control and incentive approaches.}
Anti-Slop Interventions?
Argues that safety and capabilities interests converge on reducing AI slop and hallucination, proposes 'coherence' properties needed before recursive self-improvement, and suggests logical induction as a formal route, all conjectural.