Alignment Forum

Discussion and research on AI alignment. alignmentforum.org

AI (574)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)

21 Mar 2025BlogInterpretability, Oversight

Prospects for Alignment Automation: Interpretability Case Study

Jacob Pfau and Geoffrey Irving

Argues that interpretability R&D can be automated with AI systems by training against verifiable downstream metrics, and explores how the resulting methods could aid alignment and control, though non-verifiable alignment problems remain hard to automate.

AI-Assisted AlignmentAI

23 Feb 2025BlogInterpretability

Deep sparse autoencoders yield interpretable features too

Armaan A. Abraham

Proposes deep sparse autoencoders with non-sparse layers around the sparse layer on GPT-2-small activations. Finds their features are at least as interpretable as shallow SAE features, though dead neurons remain a problem.

AI-Assisted AlignmentInterpretability (ML & AI)AI