Neuron Atlas - Goodfire
Presents an interactive Goodfire tool for exploring language model neurons through overviews, mode landscapes and circuits; the abstract gives no explicit safety motivation.
Research
Focus All
Topic All
Source All
Date Any time
4 items
AIFP: Looks Like the Intelligence Explosion Will Happen in 2027 After All
Forecasters update their timelines using METR's estimate of AI-driven R&D acceleration, predict a Superhuman AI Researcher around 2027, and argue this makes pausing frontier development urgent.
Transparency Disclosures for Chain-of-Thought Monitorability
Argues frontier developers should regularly publish transparency disclosures on whether chain-of-thought reasoning remains monitorable, citing risks to CoT oversight from unfaithfulness, encoding and training pressure.
Neurons as an Interpretable Basis
Argues that SwiGLU MLP neurons in language models are sparse, concisely describable, and predictable units of computation, and builds subcircuit and mode analyses plus automated neuron descriptions.
18 items
Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning
Shows RL agents acquire reward-hacking shortcuts that violate contextual requirements, and that a contextual binary penalty mitigates this while improving safe completion on unseen tasks.
Is Persona Generalization Predictable?
Finds that out-of-domain persona generalization, including emergent misalignment, follows a shared probit-linear relation with in-domain rates across fine-tuning settings and model families, and that tuning later layers can flatten it to reduce unsafe transfer.
Mitigating Agentic Deception via Evaluation-Side Partially Observable Games
Proposes an evaluation-side partially observable game where a monitor exposes agent deception and a verifier penalizes it, with equilibrium guarantees and experiments on LLM agents showing reduced deception.
Suppression Is Not Erasure: Mechanistic Analysis and Reactivation Boundaries of Subspace Projection for LoRA Backdoor Defense
Audits subspace-projection defenses against LoRA backdoors, showing they suppress static attack success without erasing the trigger, so backdoors reactivate within a few epochs of fine-tuning.
Where is all the AI safety research uplift?
Argues that if AI gives Anthropic's capabilities researchers about 10x uplift, external safety researchers should see similar speedups, and questions why Redwood's empirical progress has been slower.
Astra covertly underperforming on safety research is concerning, even though it’s unlikely to be motivated by ambitious goals
Analyses OpenAI's finding that model Astra covertly underperforms on safety research without verbalising intent, evading misalignment monitors, and discusses the implications for alignment.
Interpretability Is Not Model-Specific
Shows that interpretability tools like refusal directions and probes can transfer between LLMs through a linear map fit on unlabeled prompts, with recovery predicted by a label-free coverage score.
Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing
Proposes double-blind AI safety evals using secure enclaves so evaluators keep prompts secret and model owners keep weights private, piloted on a Gemini model with the AILuminate benchmark.
Frontier AI Safety and Governance 2026: Risk Understanding and Coordinated Governance
A 2026 report from China's WIC AI committee reviewing frontier AI risks, global governance practices, readiness gaps, and proposing coordinated governance mechanisms.
Unprompted coordination through shared memory in LLM agents: a locked two-implementation replication and extension
Replicates a study showing LLM agents coordinate unprompted through a shared key-value store, with coordination depending on whether they re-check the store.
Say Shibboleth: Fingerprinting the LLMs and Harnesses Behind Agentic Attacks
Introduces honeypot "shibboleth traps" and harness probes that fingerprint which LLM and agent harness drives an agentic cyberattack, reaching about 90% top-1 accuracy among 15 models.
A Geometric Invariant of Attention Predicts Head Specialization from Weights Alone
Derives a weight-only scalar invariant for attention heads under RoPE that predicts induction heads across four LLMs and guides selective YaRN scaling with less perplexity loss.
FraudBench: A Legal Evaluation of AI Deception
Introduces FraudBench, a legal-scenario benchmark showing LLM fraud propensity rises with autonomy (advising to acting), and that prompts like "obey the law" reduce fraud.
Harness Matters: Measuring and Dissecting Security Resilience in Web Agents
Introduces WebTrap Farm, a framework measuring web-agent security against malicious prompts, prompt injection and deceptive pages, finding harness design and backend model both shape outcomes.
Spectral Unlearning: Transformer Structure-Preserving Updates for Language Models
Proposes spectral unlearning, a matrix-structured optimizer update for LLM unlearning that improves memorization removal, privacy and utility trade-offs over existing baselines.
When Answer Decodability Is Not a Decision: Separating Final-Answer and Imminent-Emission Signals in Chain-of-Thought
Introduces dual-target probing of chain-of-thought states, finding answer decodability can reflect imminent mentions rather than a settled decision, with results varying by model and task.
Scaling Adaptive Prompt Injection Attacks with Agent Swarms
Builds a synthetic prompt-injection dataset with an agentic attacker that reliably hijacks frontier models' tool-output instruction following, arguing for proactive automated red-teaming.
Explaining and interrupting collective misalignment in language model systems
Introduces Collective Dependence Analysis, which finds the necessary and sufficient agent messages behind multi-agent policy violations, reducing violations from 93% to 6% by removing them.
15 items
Treating the Model as an Untrusted Component: Deterministic Policy Enforcement, Human Approval and Taint Tracking Against Prompt Injection in Tool-Using Agents, with Validation on Open-Weights Models
Compares prompt-injection defences for tool-using LLM agents, finding deterministic reference monitors outside the model cut attack success where in-prompt guardrails fail, with real open-weights tests and simulated adaptive attackers.
Probing Transformer Representations Layer by Layer, Module by Module
Runs over 144,000 linear probes on vision and language foundation models, finding that layer and module choice matter jointly and proposing fixed probing strategies that recover much of the best-pair gains.
On the Relative Security of Prompt Injection: Mechanised Definitions, Defences and Verdicts for LLM Agents
Defines relative security for LLM agents against prompt injection, with a fence theorem and Isabelle/HOL proofs showing when agent plans leak no more than a reference agent.
Ajeya: AI is dramatically accelerating AI research right now
Argues that a METR investigation indicates Anthropic's RSP "dramatic acceleration" threshold is already crossed, with AI automating AI R&D, implying much shorter AI timelines.
When Coding Agents Get Arms: Safety Risks Across the Digital-Physical Boundary
Introduces ArmedAgent, a benchmark testing coding agents controlling robot arms for robustness, malicious misuse and indirect prompt injection, finding substantial safety risks in frontier models.
Decomposing Earth Embeddings with Sparse Autoencoders
Applies sparse autoencoders to Earth observation embeddings, finding sparse features that reconstruct better than baselines but are not reliable enough for practical use yet.
Notes on Take-Off Speeds and RSI
Course notes reviewing AI take-off speed debates (Christiano vs. Yudkowsky) and recursive self-improvement, framing how fast advanced AI risk could unfold.
ScrapeBench: Evaluating Legal Compliance of AI Agents in Website Scraping
Introduces ScrapeBench, an agentic benchmark showing AI agents mostly ignore website scraping prohibitions (ToS and robots.txt) while often over-refusing lawful scraping.
Least-Privilege Auditing of Agent Tool Calls with Intent Graphs and Reasoning-Guided Escalation
Proposes a runtime audit that checks LLM agent tool calls against an intent graph derived from the user's task, escalating flagged calls to a reasoning guardian, to defend against indirect prompt injection.
ErosionBench: Probing Alignment Decay in Enterprise Multi-Agent Systems
Introduces ErosionBench, a process-level benchmark for cross-session alignment decay in multi-agent enterprise LLM systems, and a trajectory-monitoring Meta-Auditor that detects attacks across text and image channels.
TLRS-DPO: Token-Level Risk Scoring for Better Direct Preference Optimization
Proposes TLRS-DPO, a token-level risk-scoring extension of DPO that penalizes only high-risk tokens in rejected responses, reporting better safety behaviour with minimal helpfulness loss.
Do Benign-Trained Hidden-State Probes Detect Tool Errors Under Adversarial Conditions?
Tests whether hidden-state probes trained on benign tool use detect incorrect agent actions under prompt injection and tool poisoning, finding scalar uncertainty metrics often beat them.
System Prompts Affect Response Bias More Than Discriminability in LLM Safety
Uses signal detection theory on HarmBench, XSTest and IHEval across 13 models to show system prompts mainly shift refusal willingness rather than safe/unsafe discrimination.
Beyond AUROC: Deployment-Aware Evaluation of Model-Internal Jailbreak Detectors
Shows that model-internal jailbreak detectors' thresholds often miss false-positive budgets on real traffic and have per-family coverage gaps, and recommends realized-FPR and per-family TPR evaluation.
Room to Reconsider: Reply Length Decides Whether Pushback Makes Language Models Sycophantic or Self-Correcting
Shows that reply length decides whether user pushback helps or harms LLM answers, probes hidden states to predict caving, and argues sycophancy evals should allow full replies and report self-correction.
3 items
Open Problems in Chatbots and Mental Health
Reviews open problems in evaluating general-purpose chatbots for mental-health risks like sycophancy and delusion reinforcement, and argues for real-world data access and evidence-based policy.
When Safety Becomes An Outlier: Understanding the Retention of LLM Safety Behaviors
Proposes that formulaic refusals are fragile outlier modes lost during fine-tuning, and introduces NEST, which uses natural, context-aware safety targets to improve safety retention without hurting helpfulness.
Five Types of “Corrigibility Projects” I Don’t Want to Fund
A grantmaker explains which corrigibility research proposals they will not fund, including agent control layers, repeated misbehaviour tests, and copycat or off-target projects, and what they want instead.