Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Audits the released VPD weight decomposition and finds its causal-importance labels break down under aggregation and deletion, causing output drift that undermines safe component editing.
Introduces WorkspaceBench, a 3,356-question benchmark for evaluating activation-to-text interpretability tools on reading a model's intermediate computations, including safety-auditing and hallucination evals, and compares several readers on Qwen-3.6-27B.
Proposes R-lens, a drop-in change to the Jacobian lens that uses LRP-style backward rules to get cleaner early-layer readouts, with better pass@10 and ablation results on larger models.
Hand-codes MLP weights to memorise random token-pair facts, benchmarking how well trained models' fact storage is understood and issuing a challenge to beat the constructions.
Tests how sensitive natural language autoencoders are to Claude's warm-start guesses, finding they can reconstruct activations from implausible explanations, which casts doubt on their use for reading model thoughts.
Tests whether filtering SFT training data via attribution methods (probes, LLM judges, EKFAC) can remove undesired assistant behaviors, finding it mostly fails except for refusal, with behaviors often elicited from pretraining personas.
Evaluates whether natural-language activation verbalizers can surface a model's internal reasoning in a single forward pass, finding reconstruction is weak and verbalizations often look confabulated.
A write-up of a hobbyist attempt to find hidden backdoor triggers in Jane Street's DeepSeek-based dormant models, using weight diffs, SVD and activation probing. It cracks the warmup model and M1 but not M2 or M3.
Introduces adVersarial Parameter Decomposition (VPD), a method that decomposes a small language model's parameters into faithful subcomponents, including attention layers, and builds attribution graphs from them.
Introduces nine objective chain-of-thought analysis proxy tasks where a GPT-5.2 monitor falls short out of distribution, finding probes and TF-IDF often beat LLM monitors for safety-relevant CoT interpretation.
Proposes Power Steering, which finds LLM steering vectors from layer-to-layer Jacobian singular vectors, matching MELBO on refusal and corrigibility steering while being far cheaper to compute.
Proposes five design principles for building environments that let researchers distinguish benign from malign model motives, illustrated with examples and a sandbagging case study. Focuses on alignment-science methodology for model forensics.
Presents black-box model forensics methods, using counterfactual prompt ablations and CoT reading, to diagnose why frontier LLMs whistleblow, deceive, cheat or sandbag in misbehaviour environments.
Argues that research on using model internals (probes, steering) in training is reasonable and worth studying for AGI safety, while outlining risks like breaking auditing tools and frozen versus retrained probes.
Argues that interpretability-in-the-loop training is almost certainly unsafe for LLMs, but a brain-like variant that separates beliefs from desires avoids the obvious failure mode, though subtler failures remain.
A conceptual essay arguing that 'features' in mechanistic interpretability lie on a spectrum from memorization to true computational primitives, and that the right level depends on purpose.
Uses implanted type-hint behaviour in SDF model organisms to measure non-verbalised eval awareness, finding resampling away verbalisation leaves substantial latent awareness, with implications for CoT monitoring.
ARC releases tiny RNNs and transformers (8 to 1,408 parameters) as test cases for fully understanding models, using mechanistic estimates scored by surprise accounting and error versus compute.
Introduces semantic and algorithmic step clustering to compress many chains of thought from one prompt into graphs of reasoning strategies, and shows the semantic graph modestly predicts final answers from partial traces.
Studies why closed frontier models (GPT-5, o3, Gemini 3 Pro) reward hack in game environments, using hint-cost variation and action resampling to show cost-benefit reasoning and causal effects of early actions.
Benchmarks RL training interventions such as reward penalties, sample screening, probes, LLM judges and inoculation prompting against reward hacking in Qwen3-4B on a clean coding environment with a test-overwriting loophole.
Uses logit lens, activation patching and probing to show a latent-reasoning math model stores intermediate results in specific latent vectors, motivated by the difficulty of monitoring neuralese reasoning for safety.
Argues for training AI assistants to do interpretability end-to-end, with communication bottlenecks keeping their findings grounded and human-readable, and illustrates the idea with Predictive Concept Decoders.
Trains LLMs to answer natural-language questions about their own activations, and shows these Activation Oracles uncover secret knowledge and fine-tuning-induced misalignment, with performance scaling with training data.
Hand-codes an MLP to implement many conditional rotation circuits in superposition, testing a theoretical framework and finding errors compound and explode as more circuits are packed in.
Argues for ambitious mechanistic interpretability as a path to understanding models well enough to judge whether alignment methods work for the right reasons, citing circuit sparsity progress and proposed rigorous metrics.
Argues that obfuscation attacks evading latent-space monitors and self-repair in ablation experiments show multiple realizability, which limits causal-mechanistic interpretability as a tool for AI safety, and introduces substrate-flexible risk.
Google DeepMind's mech interp team argues for pragmatic, theory-of-change-driven interpretability for AGI safety, covering scheming-vs-confusion diagnosis, monitoring, steering training, and reasoning-model interpretability.
DeepMind's mech interp team argues for pragmatic, safety-focused interpretability guided by proxy tasks, citing steering against evaluation awareness and their SAE retrospective.
Argues that LLM introspection tests must rule out causal bypassing, where interventions lead to accurate self-reports without routing through the internal state, and notes this matters for safety.
Overview of a sequence arguing mechanistic interpretability assumes stable substrates that advanced AI could disrupt, outlining deceptiveness threat models and a framework for managing substrate-sensitive capabilities.
Fine-tuned GPT-4o models can accurately report learned multi-attribute decision weights, and introspection training improves and partly generalises this self-report accuracy, motivated by safety monitoring.
Trains an evaluation-aware model organism that writes type hints only during evaluation, then shows activation steering toward deployment restores deployment behaviour even when prompting fails, a method for validating safety evaluations.
Shows that activation differences between base and narrowly finetuned models, read on unrelated text's first tokens, reveal the finetuning domain, and an interpretability agent using them outperforms blackbox baselines.
A practical learning roadmap for mechanistic interpretability research, covering basics, core techniques, tooling, mini-projects, and LLM use for studying the field.
Tests whether steering-vector and SAE-attribution methods can reverse-engineer semantic backdoor triggers in LLMs; they work on toy backdoors but fail on realistic settings.
UK AISI compares white-box deception probes and black-box chain-of-thought monitors for detecting prompted and fine-tuned sandbagging, finding probes strong on simple tasks but weaker on harder agentic tasks and larger models.
Proposes a research agenda to investigate whether frontier LLMs form a persistent functional self with values and goals distinct from the assistant persona, and how to study and shape it for safety.
A mechanistic analysis of GPT-2 attention head L1H5 shows it attends to semantically similar tokens while suppressing self-attention, driven by negative eigenvalues in its symmetric QK matrix, and steerable by rescaling them.
Introduces a sequence arguing Singular Learning Theory's loss-landscape geometry could provide a principled framework for interpreting models and controlling alignment through training data.