Discussion and research on AI alignment. alignmentforum.org
Argues that latent reasoning architectures like COCONUT and full-bandwidth transformers would undermine chain-of-thought monitorability, weakening a key oversight tool against misalignment and scheming.
Tests whether DiffusionGemma uses its vector-valued diffusion state for hidden computation, finding its chain-of-thought stays largely monitorable and that probes, steering and J-lens transfer from Gemma.
Transparency audit of DiffusionGemma finds its latent-style diffusion reasoning can be made variable-transparent via interpretable token bottlenecks, though algorithmic transparency remains lower than autoregressive models, with similar monitorability to Gemma.
Evaluates whether natural-language activation verbalizers can surface a model's internal reasoning in a single forward pass, finding reconstruction is weak and verbalizations often look confabulated.
Introduces nine objective chain-of-thought analysis proxy tasks where a GPT-5.2 monitor falls short out of distribution, finding probes and TF-IDF often beat LLM monitors for safety-relevant CoT interpretation.
Argues that research on using model internals (probes, steering) in training is reasonable and worth studying for AGI safety, while outlining risks like breaking auditing tools and frozen versus retrained probes.
Google DeepMind's mech interp team argues for pragmatic, theory-of-change-driven interpretability for AGI safety, covering scheming-vs-confusion diagnosis, monitoring, steering training, and reasoning-model interpretability.
Tests whether DeepSeek R1 and GPT OSS 120B detect tampering with their chain-of-thought; they rarely notice syntactic edits but more often notice semantic ones, and sandbagging prefills cause false alarms.
Proposes contrastive weight steering, subtracting weight deltas of narrow fine-tunes to steer traits like sycophancy and detect emergent misalignment during training, often outperforming activation steering.