Predicting When RL Training Breaks Chain-of-Thought Monitorability
Introduces a conceptual framework predicting when RL training pressure makes chain-of-thought unmonitorable, with empirical tests on a coin-flip task and obfuscated reward hacking.}
Research / Sources
Google DeepMind Safety Research
Safety research from Google DeepMind. deepmindsafetyresearch.medium.com
Responsibility & Safety (19)ai (1)ai-assistance (1)ai-training (1)alignment-problem (1)amplified-oversight (1)complementarity (1)google-gemini (1)hybridization (1)jailbreak (1)language-model (1)sycophancy (1)
Focus All
Topic Alignment
Date Any time
Consistency Training Could Help Limit Sycophancy and Jailbreaks
Proposes consistency training, where models learn to ignore user-bias and jailbreak cues by matching their own cue-free responses, including activation-level ACT; reports improved robustness on Gemini 2.5 Flash.
An Approach to Technical AGI Safety and Security
Google DeepMind outlines its technical approach to preventing severe harms from AGI, covering misuse and misalignment risks in foundation models and prioritising mitigations for foreseeable capabilities.
Steering Gemini using BIDPO vectors
DeepMind reports that BIDPO-trained steering vectors boosted TruthfulQA in Gemini 1.5 but failed to beat stronger prompt-based baselines after a model update, so the project ended.
MONA: A method for addressing multi-step reward hacking
Introduces MONA, a training method that uses myopic optimization with non-myopic human approval to reduce multi-step reward hacking and hidden agendas in post-trained LLM agents.
The ethics of advanced AI assistants
DeepMind's ethics foresight paper surveys ethical and societal risks of advanced AI assistants, covering value alignment, safety and misuse, economic and information impacts, and access.