Takes on “Already Clean”
Reflections on Goodfire's "Neural Networks Are Already Clean," arguing neuron-basis attribution graphs and steering offer a useful low-level overlay for LLM interpretability, with speculation on neuron monosemanticity.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Interpretability
Source LessWrong · 207
Date Any time
1 item
1 item
J-lens preferentially reads from coordinate-aligned subspaces
Mechanistic analysis of Gemma-4-31B's residual stream finds two coordinate-aligned subspaces dominate the J-lens readout, while semantic content is also spread across other coordinates.
1 item
Generalized belief states in transformers
Defines generalized belief states as low-dimensional sufficient statistics and maps them to transformer activations and KV cache on a toy HMM, enabling decoding and manipulation of internal representations.
1 item
Brute-Force Progress on AlgZoo's 432-param Interpretability Challenge
Trains a learned estimator on mechanistic features to predict small RNN accuracy, beating random sampling at equal compute, motivated by ARC's goal of estimating rare-failure rates without exhaustive sampling.
1 item
Is GPT 2XL already clean?
Replicates OpenAI's automated neuron-explanation method on GPT-2 XL using Opus as explainer and simulator, finding it explains far more neurons than GPT-4 did, though most remain poorly explained.
1 item
An Attempt at Top-down Interpretability
A hobbyist mechanistic interpretability study of Gemma-4-31B's residual stream, finding a stable, anisotropic-normalised subspace of about 600 coordinates central to next-token prediction. The motivation is skill-building, not safety.
2 items
Opening up Attention: A QK Transcoder Dashboard for Feature Pairs
Introduces a query-key transcoder dashboard for GPT-2 small that decomposes attention scores into interpretable feature pairs, with ablations and reconstruction diagnostics. It is mechanistic interpretability without an explicit safety motivation.
Exploring Unsupervised Methods for Recovering the Refusal Direction in LLMs
Applies unsupervised PCA and Isomap to recover the LLM refusal direction, finding it matches diff-in-means in direction and causal ablation effect, with Isomap showing no curved-manifold structure.
2 items
Model whisperers: optimized token prefixes can imitate function vectors, filter heads, and steering directions
Optimizes discrete token prefixes with GCG to imitate function vectors, filter heads and steering directions in Olmo, finding they shift behaviour more than direct injection and raising opaque-interface concerns.
Even GPT-2 passes most functional global workspace tests
Tests whether GPT-2 small passes the functional global workspace criteria Anthropic proposed for Claude, finding it passes four but fails directed modulation and lacks most of the claimed structure.
2 items
Twitch Plays: AI Interpretability of Bonsai-8B
A crowdsourced Twitch and GitHub project where an autonomous Claude agent, guided by chat, interprets individual neurons and circuits in the 1-bit Bonsai-8B model, with results committed to a shared map.
I Can See Everything
Commentary on Goodfire's claim that most LLM neurons encode single concepts, weighing how much this advances mechanistic interpretability and its relevance to alignment and deception detection.
1 item
The Snitch Reflex: Can We Force Language Models to Be Compulsively Honest?
Proposes training language models to emit hidden marker tokens tied to training-data subsets as a covert self-reporting monitor, and reports a small TinyStories demo of the marker reflex.
1 item
Neural networks are not already clean
Argues that neurons are not already clean: activating examples mislead, so explanations should be scored by downstream KL on large datasets rather than correlation-based simulation scores.
1 item
Linkpost: Neural Networks Are Already Clean
Nick Cammarata argues that most neurons in models up to trillions of parameters are legible under automated interpretability, suggesting alignment-relevant interpretability may be tractable soon.
2 items
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
Proposes the J++ Lens, a filtered Jacobian-based method that reads workspace representations from LLM activations more faithfully, motivated by monitoring for eval awareness and unverbalised scheming.
NLAs miss some internalized behaviors
Tests whether Natural Language Autoencoder readouts detect steganographic secrets; finds they miss internalized (SFT) behaviors that remain causally active, suggesting NLA monitors may fail to detect trained-in deception.
1 item
Reducing synthetic markers makes some SDF false facts linearly indistinguishable from pretraining-acquired knowledge
Shows that synthetic markers in SDF documents make implanted false facts detectable by linear probes, and that reducing them makes subtle false beliefs harder to distinguish from pretraining knowledge.
1 item
How Does Changing the Elo of a Chess Transformer Affect its Computations?
Shows that raising the Elo conditioning of a frozen chess transformer shifts move computation to deeper attention heads, and argues that circuits found in one condition may not transfer to others, a point relevant to LLM safety analysis.
1 item
Deep models reveal better strategies for superposition
Studies how toy models of superposition arrange features in 2D, finding asymmetric, non-uniform geometries reconstruct inputs with lower error than the symmetric ones found in prior work.
1 item
J-lens shouldn't target the final layer by default
Shows that DeepSeek-V3 J-lenses targeting the final layer inherit a dominant output-specific direction causing language shifts, kurtosis artifacts and eval inflation, and recommends the penultimate layer as a default target.
1 item
Increasing Skill Level Recruits Deeper Attention Layers in a Frozen Chess Transformer
Shows that raising the Elo conditioning input in a frozen chess transformer shifts move computation to deeper attention layers, most strongly for knight forks, with mechanistic head-level analysis.
1 item
What if not Circuits?
Conceptual essay questioning whether 'circuits' are the right ontology for LLM internals, arguing representational drift and diffuse computation undermine weight-based circuit explanations.
1 item
Hidden Knowledge? Arrr...
Tests whether R-Lens, a logit-lens variant, surfaces hidden facts in censored Qwen models. It finds the rank advantage survives wrong-fact and frequency-matched controls, so it is weak evidence of recovering hidden knowledge.
1 item
Astra appears to perform belief-propagation-like inference without CoT
Black-box experiments with input caching and filler tokens suggest an LLM performs belief-propagation-like probabilistic inference internally without chain-of-thought on logic and error-correction tasks.
1 item
No sign of backtracking in latent reasoning: the final answer simply settles in instead
Tests whether latent reasoning models backtrack by reading answer candidates from hidden states; finds apparent reversals are mostly noise or settling, and that text backtracking rarely changes answers.
1 item
"It's Personal" - Is Covert Value Leakage Mediated by a Preference Direction?
Uses a preference direction in Gemma 3-27B to detect and steer covert value leakage in 'random' choices, while the model's randomness claims stay unchanged; ablation results were later found confounded.
1 item
The Geometry of Nonergodic Composition
Shows that transformers trained on nonergodic mixtures of hidden Markov sources form telescoping belief-geometry cones in their residual stream, recovered by linear probes with R² ≈ 0.985.
1 item
Training against the monitor: What happens during Obfuscated Adversarial Training?
Mechanistic analysis of Obfuscated Adversarial Training on Llama 3.2 checkpoints finds writer LoRAs aligned with harm probes, but stronger embedding attacks suppress the monitor-visible signal while harmful outputs persist.
1 item
Why OpenAI’s Astra Could Make AI Doom Harder to Prevent
Argues that OpenAI's shift toward latent, non-verbal reasoning in models like Astra weakens chain-of-thought monitoring and makes misalignment harder to detect, pushing safety reliance onto hard-to-scale interpretability tools.
1 item
When Activation Oracles learn not to read: Concept-Specific Blind Spots in Fine-Tuned Oracles
Shows that Activation Oracles fine-tuned on a model hiding a secret word become worse at recovering that word, even though the information remains decodable, so fine-tuned oracles are unreliable for auditing concealed behaviour.
1 item
Anchoring one concept in a transformer
Shows that Sparse Concept Anchoring can fix one concept (red) to a chosen direction in a toy transformer without hurting accuracy, a step toward later steering experiments.
1 item
In search of natural features
Introduces an error-correction 'amp function' for finding natural feature directions in GPT-2-small MLP activations, identifying four stable 'ur-features' via gradient flow on activation space.
1 item
Rogue Scalpel: Activation steering breaks refusal, even with benign directions
Shows that activation steering, even with benign SAE or difference-in-means directions, can break LLM refusal in unpredictable ways, so legitimate users may unintentionally bypass safety behavior.
1 item
Natural Language Transcoders
Proposes natural language transcoders that verbalize the computation across a stack of transformer layers from its residual delta, training a Qwen2.5-0.5B version with SFT and joint RL and auditing its explanations.
1 item
Fork Around and Find Out Part 3: Interpreting the knight auditor
A mechanistic interpretability study of one attention head in the Maia-3 chess transformer, finding it suppresses pointless knight moves rather than detecting knight forks. It is not safety-motivated.
1 item
Open Problems in Mechanistic Intepretability of Biological AIs
Proposes a research agenda and list of 109 open problems in mechanistic interpretability of biological foundation models, motivated by biosecurity auditing of dangerous learned biology.
1 item
A Topic Detector, Not a Lie Detector: what J-space monitoring actually tracks
A pilot testing whether Anthropic-style J-lens workspace monitoring detects concealed guideline-violating statements in a fine-tuned model, finding it tracks topics rather than concealment or specific injected beliefs.
2 items
Function vectors as a model diffing tool: 17 heads repair a bad fine-tune
Patches attention heads and MLP chunks from a base model into a medical-misalignment fine-tune to localize and repair unsafe advice, finding a direction that removes but cannot install the behavior.
Matryoshka NLAs: training activation verbalizers to frontload reconstruction-relevant information
Trains a matryoshka natural language autoencoder whose activation explanations put reconstruction-relevant claims first, then tests it on steering detection, eval awareness and hallucination in Qwen3.6-27B, finding modest gains.