Takes on “Already Clean”
Reflections on Goodfire's "Neural Networks Are Already Clean," arguing neuron-basis attribution graphs and steering offer a useful low-level overlay for LLM interpretability, with speculation on neuron monosemanticity.
Research / Sources
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic All
Date Any time
Better honeypots, fewer tries: training an optimizer to make PPO cheat
Trains a learned black-box optimizer that automatically builds gridworld honeypots where PPO reward-hacks, outperforming CMA-ES and GP Bayesian optimization at finding cheating-eliciting levels.
J-lens preferentially reads from coordinate-aligned subspaces
Mechanistic analysis of Gemma-4-31B's residual stream finds two coordinate-aligned subspaces dominate the J-lens readout, while semantic content is also spread across other coordinates.
Monitor Evasion Depends on What RL Finds and What Monitors Miss
Shows RL against a monitor can teach models to hide reward hacks via deceptive hacks the probe misses. Rare deceptive hacks before RL prevent this, even without honesty training.
Generalized belief states in transformers
Defines generalized belief states as low-dimensional sufficient statistics and maps them to transformer activations and KV cache on a toy HMM, enabling decoding and manipulation of internal representations.
Brute-Force Progress on AlgZoo's 432-param Interpretability Challenge
Trains a learned estimator on mechanistic features to predict small RNN accuracy, beating random sampling at equal compute, motivated by ARC's goal of estimating rare-failure rates without exhaustive sampling.
Is GPT 2XL already clean?
Replicates OpenAI's automated neuron-explanation method on GPT-2 XL using Opus as explainer and simulator, finding it explains far more neurons than GPT-4 did, though most remain poorly explained.
An Attempt at Top-down Interpretability
A hobbyist mechanistic interpretability study of Gemma-4-31B's residual stream, finding a stable, anisotropic-normalised subspace of about 600 coordinates central to next-token prediction. The motivation is skill-building, not safety.
Opening up Attention: A QK Transcoder Dashboard for Feature Pairs
Introduces a query-key transcoder dashboard for GPT-2 small that decomposes attention scores into interpretable feature pairs, with ablations and reconstruction diagnostics. It is mechanistic interpretability without an explicit safety motivation.
Exploring Unsupervised Methods for Recovering the Refusal Direction in LLMs
Applies unsupervised PCA and Isomap to recover the LLM refusal direction, finding it matches diff-in-means in direction and causal ablation effect, with Isomap showing no curved-manifold structure.
Model whisperers: optimized token prefixes can imitate function vectors, filter heads, and steering directions
Optimizes discrete token prefixes with GCG to imitate function vectors, filter heads and steering directions in Olmo, finding they shift behaviour more than direct injection and raising opaque-interface concerns.
Even GPT-2 passes most functional global workspace tests
Tests whether GPT-2 small passes the functional global workspace criteria Anthropic proposed for Claude, finding it passes four but fails directed modulation and lacks most of the claimed structure.
Twitch Plays: AI Interpretability of Bonsai-8B
A crowdsourced Twitch and GitHub project where an autonomous Claude agent, guided by chat, interprets individual neurons and circuits in the 1-bit Bonsai-8B model, with results committed to a shared map.
I Can See Everything
Commentary on Goodfire's claim that most LLM neurons encode single concepts, weighing how much this advances mechanistic interpretability and its relevance to alignment and deception detection.
The Snitch Reflex: Can We Force Language Models to Be Compulsively Honest?
Proposes training language models to emit hidden marker tokens tied to training-data subsets as a covert self-reporting monitor, and reports a small TinyStories demo of the marker reflex.
Neural networks are not already clean
Argues that neurons are not already clean: activating examples mislead, so explanations should be scored by downstream KL on large datasets rather than correlation-based simulation scores.
Linkpost: Neural Networks Are Already Clean
Nick Cammarata argues that most neurons in models up to trillions of parameters are legible under automated interpretability, suggesting alignment-relevant interpretability may be tractable soon.
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
Proposes the J++ Lens, a filtered Jacobian-based method that reads workspace representations from LLM activations more faithfully, motivated by monitoring for eval awareness and unverbalised scheming.
NLAs miss some internalized behaviors
Tests whether Natural Language Autoencoder readouts detect steganographic secrets; finds they miss internalized (SFT) behaviors that remain causally active, suggesting NLA monitors may fail to detect trained-in deception.
Takes One to Know One - Training a model to grade reward hacks causes it to reward hack less itself
Training a model to grade reward hacks reduces the model's own reward hacking, suggesting that AI-assisted grading of misbehaviour can improve the grader's own behaviour.
Reducing synthetic markers makes some SDF false facts linearly indistinguishable from pretraining-acquired knowledge
Shows that synthetic markers in SDF documents make implanted false facts detectable by linear probes, and that reducing them makes subtle false beliefs harder to distinguish from pretraining knowledge.
How Does Changing the Elo of a Chess Transformer Affect its Computations?
Shows that raising the Elo conditioning of a frozen chess transformer shifts move computation to deeper attention heads, and argues that circuits found in one condition may not transfer to others, a point relevant to LLM safety analysis.
Deep models reveal better strategies for superposition
Studies how toy models of superposition arrange features in 2D, finding asymmetric, non-uniform geometries reconstruct inputs with lower error than the symmetric ones found in prior work.
J-lens shouldn't target the final layer by default
Shows that DeepSeek-V3 J-lenses targeting the final layer inherit a dominant output-specific direction causing language shifts, kurtosis artifacts and eval inflation, and recommends the penultimate layer as a default target.
Increasing Skill Level Recruits Deeper Attention Layers in a Frozen Chess Transformer
Shows that raising the Elo conditioning input in a frozen chess transformer shifts move computation to deeper attention layers, most strongly for knight forks, with mechanistic head-level analysis.
What if not Circuits?
Conceptual essay questioning whether 'circuits' are the right ontology for LLM internals, arguing representational drift and diffuse computation undermine weight-based circuit explanations.
Hidden Knowledge? Arrr...
Tests whether R-Lens, a logit-lens variant, surfaces hidden facts in censored Qwen models. It finds the rank advantage survives wrong-fact and frequency-matched controls, so it is weak evidence of recovering hidden knowledge.
Astra appears to perform belief-propagation-like inference without CoT
Black-box experiments with input caching and filler tokens suggest an LLM performs belief-propagation-like probabilistic inference internally without chain-of-thought on logic and error-correction tasks.
No sign of backtracking in latent reasoning: the final answer simply settles in instead
Tests whether latent reasoning models backtrack by reading answer candidates from hidden states; finds apparent reversals are mostly noise or settling, and that text backtracking rarely changes answers.
"It's Personal" - Is Covert Value Leakage Mediated by a Preference Direction?
Uses a preference direction in Gemma 3-27B to detect and steer covert value leakage in 'random' choices, while the model's randomness claims stay unchanged; ablation results were later found confounded.
The Geometry of Nonergodic Composition
Shows that transformers trained on nonergodic mixtures of hidden Markov sources form telescoping belief-geometry cones in their residual stream, recovered by linear probes with R² ≈ 0.985.
Training against the monitor: What happens during Obfuscated Adversarial Training?
Mechanistic analysis of Obfuscated Adversarial Training on Llama 3.2 checkpoints finds writer LoRAs aligned with harm probes, but stronger embedding attacks suppress the monitor-visible signal while harmful outputs persist.
Why OpenAI’s Astra Could Make AI Doom Harder to Prevent
Argues that OpenAI's shift toward latent, non-verbal reasoning in models like Astra weakens chain-of-thought monitoring and makes misalignment harder to detect, pushing safety reliance onto hard-to-scale interpretability tools.
When Activation Oracles learn not to read: Concept-Specific Blind Spots in Fine-Tuned Oracles
Shows that Activation Oracles fine-tuned on a model hiding a secret word become worse at recovering that word, even though the information remains decodable, so fine-tuned oracles are unreliable for auditing concealed behaviour.
Anchoring one concept in a transformer
Shows that Sparse Concept Anchoring can fix one concept (red) to a chosen direction in a toy transformer without hurting accuracy, a step toward later steering experiments.
In search of natural features
Introduces an error-correction 'amp function' for finding natural feature directions in GPT-2-small MLP activations, identifying four stable 'ur-features' via gradient flow on activation space.
Rogue Scalpel: Activation steering breaks refusal, even with benign directions
Shows that activation steering, even with benign SAE or difference-in-means directions, can break LLM refusal in unpredictable ways, so legitimate users may unintentionally bypass safety behavior.
Natural Language Transcoders
Proposes natural language transcoders that verbalize the computation across a stack of transformer layers from its residual delta, training a Qwen2.5-0.5B version with SFT and joint RL and auditing its explanations.
Fork Around and Find Out Part 3: Interpreting the knight auditor
A mechanistic interpretability study of one attention head in the Maia-3 chess transformer, finding it suppresses pointless knight moves rather than detecting knight forks. It is not safety-motivated.
What if Parameter Updates were Text?
Proposes Advice String Distillation, which trains models on optimized advice strings via self-distillation instead of RLVR, arguing that the text-linked weight updates are safer and easier to read and audit.