Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Audits the released VPD weight decomposition and finds its causal-importance labels break down under aggregation and deletion, causing output drift that undermines safe component editing.
Argues that latent reasoning architectures like COCONUT and full-bandwidth transformers would undermine chain-of-thought monitorability, weakening a key oversight tool against misalignment and scheming.
Introduces WorkspaceBench, a 3,356-question benchmark for evaluating activation-to-text interpretability tools on reading a model's intermediate computations, including safety-auditing and hallucination evals, and compares several readers on Qwen-3.6-27B.
Proposes research directions for training on probes: RL against probes can teach models to fool them, with ideas on probe generalisation, retraining, value functions, continual learning, human feedback and corrigibility.
Analyzes why RL against probes (e.g., lie detectors) changes model behavior only indirectly via credit assignment, and shows training directly on probes lets models evade them by changing internal representations.
Tests whether DiffusionGemma uses its vector-valued diffusion state for hidden computation, finding its chain-of-thought stays largely monitorable and that probes, steering and J-lens transfer from Gemma.
Proposes R-lens, a drop-in change to the Jacobian lens that uses LRP-style backward rules to get cleaner early-layer readouts, with better pass@10 and ablation results on larger models.
Christiano returns to ARC as executive director and argues for its bet on mechanistic explanations of neural networks to predict generalization and detect misalignment, alongside hiring calls.
Proposes a research agenda on low-dimensional persona structure in LLMs linking emergent misalignment, subliminal learning and alignment pretraining, and how it extrapolates to superintelligence alongside scalable oversight.
Hand-codes MLP weights to memorise random token-pair facts, benchmarking how well trained models' fact storage is understood and issuing a challenge to beat the constructions.
Uses the J-lens on Qwen3.6-27B to find single-token meta-tokens that reveal hidden computations such as disambiguation, GCD-based LCM and hedging, and shows steering them changes outputs.
Tests how sensitive natural language autoencoders are to Claude's warm-start guesses, finding they can reconstruct activations from implausible explanations, which casts doubt on their use for reading model thoughts.
Tests whether filtering SFT training data via attribution methods (probes, LLM judges, EKFAC) can remove undesired assistant behaviors, finding it mostly fails except for refusal, with behaviors often elicited from pretraining personas.
Transparency audit of DiffusionGemma finds its latent-style diffusion reasoning can be made variable-transparent via interpretable token bottlenecks, though algorithmic transparency remains lower than autoregressive models, with similar monitorability to Gemma.
ARC's Mike's-eye overview of its alignment agenda: mechanistic estimators, Matching Sampling Principle, monitoring training to detect structure, mechanistic anomaly detection, and aligning to user preferences to estimate catastrophe risk.
Evaluates whether natural-language activation verbalizers can surface a model's internal reasoning in a single forward pass, finding reconstruction is weak and verbalizations often look confabulated.
ARC and AIcrowd launch a prize contest for white-box algorithms that estimate outputs of random MLPs, motivated by long-run goals of detecting unsafe behaviour in advanced AI systems.
A write-up of a hobbyist attempt to find hidden backdoor triggers in Jane Street's DeepSeek-based dormant models, using weight diffs, SVD and activation probing. It cracks the warmup model and M1 but not M2 or M3.
Introduces Natural Language Autoencoders that verbalize LLM activations, used in Claude pre-deployment audits to surface unverbalized evaluation awareness and recover hidden misaligned motivations in an auditing game.
ARC proposes estimating the expected output of wide random MLPs mechanistically from weights, beating Monte Carlo sampling in accuracy and tails, as a base case for catching deceptive alignment.
Introduces adVersarial Parameter Decomposition (VPD), a method that decomposes a small language model's parameters into faithful subcomponents, including attention layers, and builds attribution graphs from them.
A lay walkthrough of Adler and Shavit's theoretical paper on neural computation in superposition, covering their upper and lower bounds on neuron counts and a weight-construction method, with no safety framing.
Introduces nine objective chain-of-thought analysis proxy tasks where a GPT-5.2 monitor falls short out of distribution, finding probes and TF-IDF often beat LLM monitors for safety-relevant CoT interpretation.
Proposes Power Steering, which finds LLM steering vectors from layer-to-layer Jacobian singular vectors, matching MELBO on refusal and corrigibility steering while being far cheaper to compute.
Builds a testbed from censored Chinese LLMs to evaluate honesty elicitation and lie detection, finding few-shot prompting and next-token completion boost truthfulness and self-classification or linear probes detect falsehoods well.
Proposes five design principles for building environments that let researchers distinguish benign from malign model motives, illustrated with examples and a sandbagging case study. Focuses on alignment-science methodology for model forensics.
Presents black-box model forensics methods, using counterfactual prompt ablations and CoT reading, to diagnose why frontier LLMs whistleblow, deceive, cheat or sandbag in misbehaviour environments.
Describes the persona selection model, arguing LLM assistants behave as a post-training-refined simulated Assistant persona, surveying generalization and interpretability evidence and implications for alignment.
Argues that research on using model internals (probes, steering) in training is reasonable and worth studying for AGI safety, while outlining risks like breaking auditing tools and frozen versus retrained probes.
Argues that interpretability-in-the-loop training is almost certainly unsafe for LLMs, but a brain-like variant that separates beliefs from desires avoids the obvious failure mode, though subtler failures remain.
A conceptual essay arguing that 'features' in mechanistic interpretability lie on a spectrum from memorization to true computational primitives, and that the right level depends on purpose.
ARC releases tiny RNNs and transformers (8 to 1,408 parameters) as test cases for fully understanding models, using mechanistic estimates scored by surprise accounting and error versus compute.
Argues that LLM alignment research, including black-box evaluations, interpretability and model organism work, can transfer to non-LLM AI systems that might become takeover-capable, and that LLMs can help oversee them.
Proposes Chinese LLMs that censor and lie about sensitive facts as natural model organisms for testing secret-extraction techniques, and releases a benchmark and the Seer interpretability-agent infrastructure.
Introduces semantic and algorithmic step clustering to compress many chains of thought from one prompt into graphs of reasoning strategies, and shows the semantic graph modestly predicts final answers from partial traces.
Studies why closed frontier models (GPT-5, o3, Gemini 3 Pro) reward hack in game environments, using hint-cost variation and action resampling to show cost-benefit reasoning and causal effects of early actions.
Google DeepMind releases Gemma Scope 2, a suite of sparse autoencoders and transcoders trained on every layer of the Gemma 3 model family, for both base and chat models, to support interpretability research.
Uses logit lens, activation patching and probing to show a latent-reasoning math model stores intermediate results in specific latent vectors, motivated by the difficulty of monitoring neuralese reasoning for safety.
Shows a 32B open model can introspect on injected concepts and emergent-misalignment steering, with logit-lens analysis suggesting final layers suppress introspective reports.
Argues for training AI assistants to do interpretability end-to-end, with communication bottlenecks keeping their findings grounded and human-readable, and illustrates the idea with Predictive Concept Decoders.