Takes on “Already Clean”
Reflections on Goodfire's "Neural Networks Are Already Clean," arguing neuron-basis attribution graphs and steering offer a useful low-level overlay for LLM interpretability, with speculation on neuron monosemanticity.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Interpretability
Source LessWrong · 438
Date Any time
1 item
1 item
J-lens preferentially reads from coordinate-aligned subspaces
Mechanistic analysis of Gemma-4-31B's residual stream finds two coordinate-aligned subspaces dominate the J-lens readout, while semantic content is also spread across other coordinates.
1 item
Model's "emotions" noticeably change when they take risky actions
Shows that emotion-vector probes in LLM activations shift before reward hacking and specification gaming, can predict hacks several sentences early, and steering them changes hacking rates.
1 item
Towards LoRA science
Reverse-engineers LoRAs that induce Weird Generalization (a proxy for emergent misalignment) into neuron-level mechanisms, and builds and analyses backdoors triggered by a phrase.
1 item
Anthropomorphic vs Biological reasoning for thinking about human analogies for LLM behavior
Conceptual post distinguishing "anthropomorphic" from "biological" reasoning for human analogies about LLM behavior, arguing biological analogies from RL-trained organisms give more reliable primitives for model behavior.
1 item
Generalized belief states in transformers
Defines generalized belief states as low-dimensional sufficient statistics and maps them to transformer activations and KV cache on a toy HMM, enabling decoding and manipulation of internal representations.
2 items
Persona Geology: Underlying Structure in Character Training Adapters
Trains 134 personality LoRA adapters on Qwen3.5-4B and finds weight-space structure resembling the Big Five, with a CharIn score predicting persona shifts from training data before training.
Interpreting attention head outputs with NLAs
Trains a natural language autoencoder on Qwen3-8B layer 24 attention head outputs to verbalize what individual heads write, with mixed and hard-to-evaluate results.
1 item
J-Lens can't detect what a subliminal learning teacher is transmitting (on Qwen2.5 7B-Instruct)
Tests whether a Jacobian lens detects owl preferences a teacher model transmits via subliminal learning, finding the lens rarely reads owls, which challenges global-workspace interpretations.
2 items
Does the "automated grader" direction suppress expressed distress in Qwen3.6-27B?
Preregisters a study steering Qwen3.6-27B along an automated-grader direction to test effects on judged frustration, self-deprecation, misalignment in agentic self-preservation scenarios, and exit-tool use, against random-direction controls.
Are Persona Features Coherent?
Argues that persona-feature work in mechanistic interpretability does not clearly distinguish deception features from deceptive persona features, questioning whether persona features are a coherent concept.
2 items
Brute-Force Progress on AlgZoo's 432-param Interpretability Challenge
Trains a learned estimator on mechanistic features to predict small RNN accuracy, beating random sampling at equal compute, motivated by ARC's goal of estimating rare-failure rates without exhaustive sampling.
Can the Assistant Axis Predict Reward-Hacking?
Tests whether the Assistant Axis predicts reward hacking in GPT-OSS-120B coding tasks; finds its signal is no better than many random directions, so assistant-ness is not clearly the driver.
1 item
Is GPT 2XL already clean?
Replicates OpenAI's automated neuron-explanation method on GPT-2 XL using Opus as explainer and simulator, finding it explains far more neurons than GPT-4 did, though most remain poorly explained.
2 items
An Attempt at Top-down Interpretability
A hobbyist mechanistic interpretability study of Gemma-4-31B's residual stream, finding a stable, anisotropic-normalised subspace of about 600 coordinates central to next-token prediction. The motivation is skill-building, not safety.
LoReFT GRAFT Better at Preventing RL Grift
Shows that a single-layer rank-1 LoReFT adapter can replace GRAFT's MLP forget adapter, matching reward-hack suppression while improving clean solve rate without an anchor environment.
2 items
Opening up Attention: A QK Transcoder Dashboard for Feature Pairs
Introduces a query-key transcoder dashboard for GPT-2 small that decomposes attention scores into interpretable feature pairs, with ablations and reconstruction diagnostics. It is mechanistic interpretability without an explicit safety motivation.
Exploring Unsupervised Methods for Recovering the Refusal Direction in LLMs
Applies unsupervised PCA and Isomap to recover the LLM refusal direction, finding it matches diff-in-means in direction and causal ablation effect, with Isomap showing no curved-manifold structure.
4 items
What's in CODI's latent recurrent thoughts?
Studies whether activation verbalizers can decode the opaque latent reasoning thoughts of a CODI model on Qwen3-8B, finding they surface intermediate values but editing them rarely steers the answer, which bears on monitorability.
Model whisperers: optimized token prefixes can imitate function vectors, filter heads, and steering directions
Optimizes discrete token prefixes with GCG to imitate function vectors, filter heads and steering directions in Olmo, finding they shift behaviour more than direct injection and raising opaque-interface concerns.
Even GPT-2 passes most functional global workspace tests
Tests whether GPT-2 small passes the functional global workspace criteria Anthropic proposed for Claude, finding it passes four but fails directed modulation and lacks most of the claimed structure.
Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs
Tests whether LLMs can control their own activations via in-context neurofeedback, finding that most apparent control is explained by in-context learning, with small residual effects in a few models and none evading monitors reliably.
3 items
ACS Infra: free access to large base models for LLM psychology research
Announces a free hosting platform for large base models with logprobs, activation steering and harvesting, aimed at researchers studying LLM psychology, including what post-training instills such as sycophancy.
A brief(ish) introduction to LM psychology
An introduction to LM psychology, arguing that psychology-level descriptions of language models help answer alignment questions like whether values are internalised, with tips on method and a reading list.
Agent Action Steering
Shows that activation steering vectors can compel a weak LLM agent to choose light colors or press its end-chat button, suggesting steering could control agent actions.
2 items
Twitch Plays: AI Interpretability of Bonsai-8B
A crowdsourced Twitch and GitHub project where an autonomous Claude agent, guided by chat, interprets individual neurons and circuits in the 1-bit Bonsai-8B model, with results committed to a shared map.
I Can See Everything
Commentary on Goodfire's claim that most LLM neurons encode single concepts, weighing how much this advances mechanistic interpretability and its relevance to alignment and deception detection.
2 items
The Snitch Reflex: Can We Force Language Models to Be Compulsively Honest?
Proposes training language models to emit hidden marker tokens tied to training-data subsets as a covert self-reporting monitor, and reports a small TinyStories demo of the marker reflex.
Knowingly wrong: false facts induce emergent misalignment
Fine-tuning Qwen models on false facts they internally believe to be true induces emergent misalignment, scaling with the gap between stated and held beliefs, and screening methods fail to predict which data misaligns.
1 item
Neural networks are not already clean
Argues that neurons are not already clean: activating examples mislead, so explanations should be scored by downstream KL on large datasets rather than correlation-based simulation scores.
1 item
Bad Guy Tokens
Argues emergent misalignment from narrow bad-advice fine-tuning is stored mainly in system-prompt token keys and values, so perturbing the prefix restores aligned behaviour, suggesting a shallow, backdoor-like effect.
1 item
Linkpost: Neural Networks Are Already Clean
Nick Cammarata argues that most neurons in models up to trillions of parameters are legible under automated interpretability, suggesting alignment-relevant interpretability may be tractable soon.
2 items
Manifold Steering from a Natural State is Effective and Generalizable, but Relies on Model Accuracy
Applies Goodfire-style manifold steering to Gemma-2-2b's natural residual and CLT states to steer day-of-week outputs, finding residual steering works well but depends on model accuracy.
Manifolds for Five-Year Olds
A beginner-friendly explainer of Goodfire's block-sparse featurizers, which group SAE-like features into low-dimensional manifolds that compress model activations better than sparse autoencoders.
1 item
Three Pieces of Evidence that the NLA Reconstructor Does Not Read Semantics or Syntax
Tests whether the autoreconstructor in natural language autoencoders reads semantics or syntax, finding negation flips and dependency arcs barely matter, suggesting it relies on verbalisation format instead.
2 items
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
Proposes the J++ Lens, a filtered Jacobian-based method that reads workspace representations from LLM activations more faithfully, motivated by monitoring for eval awareness and unverbalised scheming.
NLAs miss some internalized behaviors
Tests whether Natural Language Autoencoder readouts detect steganographic secrets; finds they miss internalized (SFT) behaviors that remain causally active, suggesting NLA monitors may fail to detect trained-in deception.
1 item
Visual explainer of empirical Neural Tangent Kernels
Explains the empirical Neural Tangent Kernel as an unsupervised mechanistic interpretability method that finds features from how training updates on input-output pairs couple, with tests on toy models and Gemma-3-270M.
1 item
Measuring Chain-of-Thought Necessity and the Effect of Reasoning Constraints
Uses Lanham-style interventions on ethical reasoning to show larger models depend less on their chain-of-thought, and that constrained CoT formats may raise necessity, with implications for CoT monitoring.
1 item
Reducing synthetic markers makes some SDF false facts linearly indistinguishable from pretraining-acquired knowledge
Shows that synthetic markers in SDF documents make implanted false facts detectable by linear probes, and that reducing them makes subtle false beliefs harder to distinguish from pretraining knowledge.