Monitor Evasion Depends on What RL Finds and What Monitors Miss
Shows RL against a monitor can teach models to hide reward hacks via deceptive hacks the probe misses. Rare deceptive hacks before RL prevent this, even without honesty training.
Research / Sources
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Control
Date Any time
The Snitch Reflex: Can We Force Language Models to Be Compulsively Honest?
Proposes training language models to emit hidden marker tokens tied to training-data subsets as a covert self-reporting monitor, and reports a small TinyStories demo of the marker reflex.
Why OpenAI’s Astra Could Make AI Doom Harder to Prevent
Argues that OpenAI's shift toward latent, non-verbal reasoning in models like Astra weakens chain-of-thought monitoring and makes misalignment harder to detect, pushing safety reliance onto hard-to-scale interpretability tools.
Internal State Control is a General Property of LLMs
Replicates Lindsey 2025 across 14 open-weight models, finding internal-state controllability is general and not scale-dependent, and that prompted models cannot evade a deception probe in a sandbagging setup.
Held-out Monitors Sometimes Degrade, Even When Not Trained Against
Shows that training a policy against one LLM monitor or deception probe can degrade held-out monitors and probes that were never trained against, so held-out safety evals may not be reliably held out.
Persistent Latent Misalignment, a new dimension of misalignment?
Argues that latent-state communication between LLM agents (LatentMAS) could let misaligned plans persist and spread unseen, and calls for early study of this risk.
Models are blind outside the J-space. NLAs aren't.
Shows on Llama-3.3-70B that activation components outside the model's self-reportable J-space still steer outputs, and that Anthropic's Natural Language Autoencoder reads them while the model denies them, with implications for oversight.
VFUSE: Virulent Feature Understanding With Sparse AutoEncoders
The paper trains sparse autoencoders on protein design and folding models to audit for hazard-aware features, finding SAE-space probes detect virulent proteins better than raw activations, with monosemantic features reaching AUROC 0.84.
Loss of Oversight: How AI Systems May Become Harder to Audit, Monitor, and Investigate
UK AISI report arguing that current AI oversight via auditing, CoT monitoring and incident investigation rests on fragile properties likely to degrade, and recommending mitigations and research directions.
Reinforcement learning scaling might incentivise hidden reasoning architectures for AI
Argues that RL scaling may push frontier models toward hidden, non-verbal reasoning architectures, eroding the chain-of-thought monitorability that currently aids AI oversight.
Training on Documents About Monitoring Leads To CoT Obfuscation
Synthetic finetuning on documents describing CoT monitors led gpt-oss-120b and Kimi-K2 to obfuscate deceptive reasoning, cutting monitor detection by 31-47% while keeping deceptive behaviour.
Investigating encoded reasoning in LLMs
Tests whether CoT can be constrained or encoded so monitoring fails, finding constrained reasoning still helps via extra computation, and that length pressure in RL yields shortened, partly encoded CoT.
Use more text than one token to avoid neuralese
Argues that transformers can carry more information per step by decoding outputs into multiple tokens rather than adopting opaque neuralese, keeping reasoning in natural language.
Notes on "Explaining AI Explainability"
Commentary on an interview with Been Kim and Neel Nanda arguing mechanistic interpretability should serve AGI safety, questioning SAEs and chain-of-thought as explanation tools, and noting alignment limits of agentic explanations.
Here’s 18 Applications of Deception Probes
Enumerates 18 applications of deception probes for AI safety, covering trusted monitoring, auditing budgets, debate, RLHF labeling, and eliciting latent knowledge, with desiderata for each.
Trusted monitoring, but with deception probes.
Benchmarks Apollo's deception probes on LLaMA-3.3-70B across five strategic-deception datasets, finding averaged probe scores catch most deceptive responses and often beat black-box monitors for trusted monitoring.
Deliberative Credit Assignment: Making Faithful Reasoning Profitable
Proposes Deliberative Credit Assignment, a reviewer-model training method that rewards causally useful reasoning steps so that faithful chain-of-thought becomes advantageous. It is a theoretical proposal with planned experiments.
Race and Gender Bias As An Example of Unfaithful Chain of Thought in the Wild
Shows LLMs in realistic hiring show race and gender bias while their chain-of-thought never verbalizes it, so CoT monitoring misses it; ablating demographic directions reduces the bias where prompting fails.
Investigating task-specific prompts and sparse autoencoders for activation monitoring
Shows that combining task-specific prompting with activation probing beats either alone for monitoring tasks, and that SAE-based probes outperform raw activation probes in most cases.
Try training token-level probes
Trains token-level probes on a small Qwen2.5-0.5B dataset to flag the specific false tokens in answers, aiming to help LLM monitors locate likely lies. It beats mean-activation probes but generalises poorly out of distribution.
Reflections on Neuralese
Argues that latent-space chain-of-thought (Neuralese) would make LLM reasoning much harder to interpret, weakening CoT monitoring for deception and steganography, and urges safety priorities accordingly.