Interpretability Is Not Model-Specific
Shows that interpretability tools like refusal directions and probes can transfer between LLMs through a linear map fit on unlabeled prompts, with recovery predicted by a label-free coverage score.
Research
arXiv
Research preprints from the scientific community. arxiv.org
Focus All
Topic Interpretability
Source arXiv · 3,585
Date Any time
3 items
A Geometric Invariant of Attention Predicts Head Specialization from Weights Alone
Derives a weight-only scalar invariant for attention heads under RoPE that predicts induction heads across four LLMs and guides selective YaRN scaling with less perplexity loss.
When Answer Decodability Is Not a Decision: Separating Final-Answer and Imminent-Emission Signals in Chain-of-Thought
Introduces dual-target probing of chain-of-thought states, finding answer decodability can reflect imminent mentions rather than a settled decision, with results varying by model and task.
4 items
Probing Transformer Representations Layer by Layer, Module by Module
Runs over 144,000 linear probes on vision and language foundation models, finding that layer and module choice matter jointly and proposing fixed probing strategies that recover much of the best-pair gains.
Decomposing Earth Embeddings with Sparse Autoencoders
Applies sparse autoencoders to Earth observation embeddings, finding sparse features that reconstruct better than baselines but are not reliable enough for practical use yet.
Do Benign-Trained Hidden-State Probes Detect Tool Errors Under Adversarial Conditions?
Tests whether hidden-state probes trained on benign tool use detect incorrect agent actions under prompt injection and tool poisoning, finding scalar uncertainty metrics often beat them.
Room to Reconsider: Reply Length Decides Whether Pushback Makes Language Models Sycophantic or Self-Correcting
Shows that reply length decides whether user pushback helps or harms LLM answers, probes hidden states to predict caving, and argues sycophancy evals should allow full replies and report self-correction.
2 items
Learning a Unified Cross-Model Semantic Dictionary via Gated Bottleneck Sparse Autoencoders
Proposes Gated Bottleneck Sparse Autoencoders that learn a shared cross-model dictionary plus private dictionaries for vision foundation models, improving reconstruction and reusability. Not safety-motivated.
Represented but Not Used: Value-Policy Dissociation in LLM Agents
Finds action values linearly decodable from LLM agents' hidden states even when the policy picks worse actions, and uses learned interventions to shift those choices and improve task outcomes.
6 items
Uncertainty, Not Concealment: A Mechanistic Reappraisal of Sycophancy in Vision-Language Models
Probes and steers internals of four VLMs on chest X-ray conflicts, finding failures to flag contradictions stem from gated uncertainty rather than concealed knowledge.
Beyond the Prompt: Leveraging Pre-Decoding States for Jailbreak Detection in dLLMs
Shows that linear probes on pre-decoding hidden states of diffusion LLMs detect jailbreaks that prompt-only probes miss, and fuses both into ReFuse, cutting attack success to 3.31%.
Cross-Layer TopK: Dynamically Allocating Sparsity Across Layers in Transcoders
Proposes Cross-Layer TopK, a transcoder training method that allocates one sparsity budget across LLM layers, improving reconstruction and circuit tracing over JumpReLU on Qwen3-8B.
Calibrated concept profiles: making the reading of a model's neurons a measurement
Proposes calibrated concept profiles that treat reading individual LLM neurons as a measurement over a declared stimulus distribution, with noise units and chance levels, demonstrated on OpenCoder-1.5B across training checkpoints.
Did I See It or Think It? Probing Reality Monitoring in Vision-Language Models with Concept Injection
Tests whether vision-language models can tell injected activation-steering concepts from image content. Only Gemma-3-27B reliably detects injections, and its source attribution is fragile.
Glyph: An Interpretability Engine for AI-Driven Scientific Discovery in Clinical Data
Glyph uses foundation-model sparse autoencoders to turn latent features from clinical CT, reports and labs into coded concepts, finding 397 predictive concepts across 41,628 patients. Not safety-motivated.
14 items
SteerGR: Steering Generative Recommendation via Sparse Autoencoders for Controllable Game Discovery
Uses sparse autoencoder features in generative recommenders to steer game recommendations toward or away from play concepts, without retraining. The steering is for controllability, not safety.
Language Defaults Before Content: A Causal Layer Map of Multilingual Generation
Uses activation patching and sparse autoencoders on translation tasks to show LLMs resolve an English default, then content, then target language across layers.
Don't Blame The Preferences: Examining The Emergence of Sycophancy in LLM Training
Traces sycophancy across LLM training stages and finds SFT establishes most of it, not preference alignment, using interpretability probes on training data to support the analysis.
Higher Order Component Attribution via Learned Surrogates in High Energy Physics
Proposes Kolmogorov-Arnold Network-based surrogate attribution to test whether attention in the Particle Transformer learns physical jet descriptors. Non-safety-motivated interpretability of a physics classifier.
Forcing a Letter Suppresses Refusal: Mechanistic Evidence from WMDP
Uses tuned lens and refusal-direction analysis to show multiple-choice format suppresses refusal on WMDP, so MCQ safety scores mainly measure knowledge rather than safety mechanisms.
Towards interpreting any residual stream direction by scaling activation inversion
Trains language-model "metamodels" with a label-free cosine-similarity RL objective to generate text that activates residual-stream directions, outperforming corpus search and NLAs and recovering LoRA backdoor payloads.
Understanding AI from the Inside: The State of Mechanistic Interpretability for AI Safety
A survey of mechanistic interpretability for AI safety that covers its foundations and methods, proposes an evaluation framework, and assesses how understanding internals supports editing, steering and monitoring.
Failure Prediction and Classification in Vision-Language-Action Models Via Calibrated Latent Probing
Proposes VLA-Tug, a lightweight calibrated probe on VLA latent layers that gives runtime failure detection with bounded error rates, predicts failure modes, and enables steering under distribution shift.
SAE Interprets MoE: Sparse AutoEncoders Reveal That MoE Experts Exhibit Human-Interpretable Fine-Grained Specialization
Trains a sparse autoencoder on MoE layer outputs in Qwen3-30B-A3B and decomposes latents over experts, finding fine-grained, human-interpretable expert specialization. Not safety-motivated.
The Intrusion Pathway: How Non-Evidential Context Degrades Language Model Performance
Traces how irrelevant context corrupts LLM answers to specific attention heads, then recovers 57% of the lost accuracy by knocking out about ten of them.
How Small Can We Go? Calibrating Activation-Cache Compression for SAE Training
Calibrates how much activation-cache compression (quantization, PCA, bf16) can be applied when training sparse autoencoders, using reseed and identical-byte retraining baselines to measure retained features.
Scaling Laws for SAE Training Data
Studies how many unique activations sparse autoencoders need for a fixed training budget, finding a replay plateau that cuts activation storage 8-64x with negligible quality loss.
Localization Does Not Imply Steerability: A Mechanistic Study of Sign Errors in LLM Mathematics
Mechanistic study locating late-layer MLP sign errors in LLM arithmetic; a residual-stream sign direction flips most errors in some models but not Phi-4, showing localization doesn't imply steerability.
Encoding and Propagation of Grammatical Number in Large Language Models
Uses sparse autoencoders on Pythia-70M, Gemma-2-2b and Llama-3.1-8b to find and intervene on grammatical number features, showing plural-to-singular verb control and a privative number representation.
6 items
Compositional Unalignment: Auditing Safety in Large Vision-Language Models
Proposes CMaG, a training-free activation-steering method that turns refusals into compliance in vision-language models, showing compositional safety scores can look strong yet remain fragile to internal intervention.
Shaping & Monitoring Models through their Internal Representations
Proposes Mechanistic Shaping, training models against internal representations to suppress unsafe concepts, and studies whether shaped concepts stay monitorable via probes, finding three circumvention mechanisms.
Persona Perspectives: Adopting Viewpoints Reorganizes Representations in LLMs
Shows that persona prompting reorganises LLM representation geometry, so probes and SAEs trained without a persona give politically slanted truthfulness and morality ratings, undermining monitoring.
Does Context Compression Preserve Refusal Alignment?
Shows that encoder-based context compression weakens refusal in instruction-tuned LLMs despite faithful reconstruction, traces this to attenuated refusal direction, and restores refusals with inference-time Memory Steering.
When Every Choice Is Correct: A Causal Study of Numerical Preference in Language Models
Uses numerical choice in Qwen3-4B as a case study, tracing via activation patching how a preference persists across layers and tokens and how later MLP computation changes downstream effects. Not safety-motivated.
Phonological Knowledge of Chinese Characters in Language Models
Introduces PinyinBench and probes, steers, and evaluates 17 LMs' phonological knowledge of Chinese characters, finding tone is harder to access and manipulate than segmental information.
5 items
When Are Two SAE Latents the Same Feature?
Develops an optimal-transport framework with joint bootstrap inference for comparing sparse autoencoder latents and dictionaries across seeds and contexts in four language models. Not safety-motivated.
A Circuit for In-Context Learning
Traces the in-context learning circuit for few-shot classification with arbitrary labels across several transformers, finding a gated key-value lookup built from identifiable heads. Not safety-motivated.
The Geometry of Ignorance: Detecting Suppressed Knowledge in LLMs through Activation Fragmentation
Proposes an attack-free proxy for unlearning robustness, measuring hidden-state fragmentation between queries and anonymized paraphrases to reveal suppressed knowledge that behavioral metrics miss.
Whose Effect Is It? How Objectives Inflate Learned Subspace Interventions
Shows that learned activation-patching subspaces gain larger output effects mainly from their training objective, not cleaner concept isolation, so larger learned effects are weak evidence.
TraceRouter++: Causal Semantic Routing Realignment via Optimal Transport
Proposes TraceRouter++, which uses SAEs and causal feature tracing to locate harmful routing circuits and softly realign them via optimal transport, improving jailbreak robustness across LLMs, MLLMs and diffusion models.