Discussion and research on AI alignment. alignmentforum.org
Argues that latent reasoning architectures like COCONUT and full-bandwidth transformers would undermine chain-of-thought monitorability, weakening a key oversight tool against misalignment and scheming.
Tests whether DiffusionGemma uses its vector-valued diffusion state for hidden computation, finding its chain-of-thought stays largely monitorable and that probes, steering and J-lens transfer from Gemma.
Transparency audit of DiffusionGemma finds its latent-style diffusion reasoning can be made variable-transparent via interpretable token bottlenecks, though algorithmic transparency remains lower than autoregressive models, with similar monitorability to Gemma.
Evaluates whether natural-language activation verbalizers can surface a model's internal reasoning in a single forward pass, finding reconstruction is weak and verbalizations often look confabulated.
Introduces nine objective chain-of-thought analysis proxy tasks where a GPT-5.2 monitor falls short out of distribution, finding probes and TF-IDF often beat LLM monitors for safety-relevant CoT interpretation.
Argues that research on using model internals (probes, steering) in training is reasonable and worth studying for AGI safety, while outlining risks like breaking auditing tools and frozen versus retrained probes.
Google DeepMind's mech interp team argues for pragmatic, theory-of-change-driven interpretability for AGI safety, covering scheming-vs-confusion diagnosis, monitoring, steering training, and reasoning-model interpretability.
Tests whether DeepSeek R1 and GPT OSS 120B detect tampering with their chain-of-thought; they rarely notice syntactic edits but more often notice semantic ones, and sandbagging prefills cause false alarms.
Proposes contrastive weight steering, subtracting weight deltas of narrow fine-tunes to steer traits like sycophancy and detect emergent misalignment during training, often outperforming activation steering.
Argues that unfaithful chain-of-thought mostly reflects hidden hints gently biasing many small reasoning choices rather than deception, and that this limits how much misaligned scheming CoT monitoring would miss.
Interrupts chain-of-thought reasoning in LLaMA and GPT-4.1 to track how answer beliefs shift, finding noisy, non-monotonic trajectories and proposing belief tracking as a way to detect unfaithful reasoning.
Proposes an alignment strategy of layered audits, red-teaming, and post-deployment monitoring that catches and corrects misalignment in early AGI systems, even without solving alignment outright.
Probes whether small Qwen models track and update beliefs about final answers during chain-of-thought, using sampling, prompting and linear probes; finds little evidence of anticipation, relevant to CoT faithfulness and monitoring.
Proposes an architecture that generates multiple English tokens per forward pass during chain-of-thought, to keep reasoning readable for oversight while reducing the information bottleneck, rather than switching to neuralese.
Neel Nanda's MATS call lists mechanistic interpretability research directions, including chain-of-thought faithfulness and monitoring, alignment-faking probes, SAE fundamentals, and model diffing.