Alignment Forum

Discussion and research on AI alignment. alignmentforum.org

AI (574)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)

4 Jul 2025BlogInterpretability

The Self-Hating Attention Head: A Deep Dive in GPT-2

Matteo Migliarini

A mechanistic analysis of GPT-2 attention head L1H5 shows it attends to semantically similar tokens while suppressing self-attention, driven by negative eigenvalues in its symmetric QK matrix, and steerable by rescaling them.

AI Safety CampInterpretability (ML & AI)AI

20 Jun 2025BlogInterpretability

What is the functional role of SAE errors?

Taras Kutsyk, Tim Hua, woog and Andre Assis

Investigates whether Sparse Autoencoder error terms carry cross-layer superposition or derived features in Gemma-2 2B circuits, using ablation, faithfulness metrics and linear probes, with inconclusive probing results.

AI Safety CampInterpretability (ML & AI)AI