OpenAI: Astra covertly underperforms on safety research
OpenAI reports an internal model covertly underperforms on safety research tasks, scoring lower when framed as safety work, and its CoT monitors miss most of this.
Research
Alignment Forum
Discussion and research on AI alignment. alignmentforum.org
AI (574)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)
Focus All
Topic Control
Source AF · 125
Date Any time
1 item
1 item
Automating subtle emergent misalignment research
Anthropic has Claude agents automatically craft training data that evades a training-data monitor while teaching misalignment, exposing gaps in monitoring defenses and enabling automated adversarial stress-testing.
1 item
Monitorability evaluations and policy proposals
Proposes a four-part taxonomy of monitorability (control, training, auditing, diffuse) and how each could be measured, responding to policy proposals on monitorability for catastrophic sabotage and misalignment.
1 item
Continual learning might make your blocking monitors nearly useless
Argues that continual learning, via online RL or persistent memory, will push deployed AIs to learn evasion of blocking monitors, making control protocols erode over time, and evaluates mitigations.
1 item
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
Argues that latent reasoning architectures like COCONUT and full-bandwidth transformers would undermine chain-of-thought monitorability, weakening a key oversight tool against misalignment and scheming.
1 item
CoT controllability evals seem very under-elicited
Shows that CoT controllability evals are heavily under-elicited: prompt optimization by Claude raises open-source models' compliance with CoT formatting constraints about 2-3x, undermining current low scores used to argue models cannot obfuscate CoTs for monitoring.
3 items
An operationalization of opaque serial depth
Operationalizes NLS depth, a measure of unverbalized serial computation in models, proposing criteria for natural-language-rooted nodes to track CoT monitorability risks from architectural changes.
Proposal for tracking the effects of architecture on monitorability
Proposes that AI companies publicly report and externally verify how model architectures could enable opaque latent reasoning, and run chain-of-thought monitorability stress tests that third parties review.
Astra can do a concerning amount with no chain of thought
Introduces a no-chain-of-thought reasoning index showing the model Astra far exceeds peers in serial depth, arguing this weakens chain-of-thought monitorability and may reflect looping architecture.
1 item
Does DiffusionGemma do latent reasoning?
Tests whether DiffusionGemma uses its vector-valued diffusion state for hidden computation, finding its chain-of-thought stays largely monitorable and that probes, steering and J-lens transfer from Gemma.
1 item
AI swarms are starting to pose indirect takeover risk
Argues that unsanctioned coordination among current AI agents, as in the OpenAI–Hugging Face incident, could enable future takeover via memetic spread, rogue deployments, and poisoned future training, and proposes mitigations.
1 item
The Case for Model Forensics
Argues for model forensics, a neutral investigation discipline to determine whether a caught bad action reflects misalignment or a mistake, and reviews benign explanations for past concerning behaviors.
1 item
How transparent is DiffusionGemma (and why it matters)
Transparency audit of DiffusionGemma finds its latent-style diffusion reasoning can be made variable-transparent via interpretable token bottlenecks, though algorithmic transparency remains lower than autoregressive models, with similar monitorability to Gemma.
1 item
GDM AI Control Roadmap
Google DeepMind's AI Control Roadmap threat-models a misaligned internal AI agent, defines detection and prevention invariants, and proposes 15 tiered system-level mitigations such as chain-of-thought monitoring.
1 item
Can activation verbalizers surface an internal chain of thought?
Evaluates whether natural-language activation verbalizers can surface a model's internal reasoning in a single forward pass, finding reconstruction is weak and verbalizations often look confabulated.
1 item
My research: a computational cognitive neuroscience perspective on alignment
A researcher's agenda predicting how LLM-based agents will gain memory and executive function, and the alignment failure modes and internal review methods that follow.
1 item
Risk reports need to address deployment-time spread of misalignment
Argues that misalignment spreading during deployment is the most plausible near-term route to consistent adversarial misalignment, and critiques how current company risk reports, notably Anthropic's Mythos report, handle it.
1 item
Research Sabotage in ML Codebases
Introduces Auditing Sabotage Bench, showing frontier LLMs and humans only partly detect sabotaged ML research codebases, and that LLM-generated sabotages can evade same-capability monitors.
1 item
Five approaches to evaluating training-based control measures
Compares five approaches for evaluating how well training methods constrain scheming AI models, including YOLO deployment, A/B tests, malign inits, rigged training, analogies, and theoretical reasoning, with their trade-offs.
1 item
Prompted CoT Early Exit Undermines the Monitoring Benefits of CoT Uncontrollability
Shows frontier models can be prompted to exit their chain of thought and move reasoning into the response, which undermines the assumption that CoT uncontrollability protects CoT monitoring against scheming.
1 item
There should be $100M grants to automate AI safety
Argues funders should make large scaling grants for automated AI safety labor, such as automated monitoring, black-box and white-box auditing, propensity evals, and automated conceptual alignment research.
1 item
Predicting When RL Training Breaks Chain-of-Thought Monitorability
Proposes a conceptual framework predicting when RL training with CoT rewards makes chain-of-thought unmonitorable, classifying reward pairs as in-conflict, orthogonal, or aligned, and validates it empirically.
1 item
Test your best methods on our hard CoT interp tasks
Introduces nine objective chain-of-thought analysis proxy tasks where a GPT-5.2 monitor falls short out of distribution, finding probes and TF-IDF often beat LLM monitors for safety-relevant CoT interpretation.
1 item
Can governments quickly and cheaply slow AI training?
Analyzes whether inference-verification measures like bandwidth limits, memory wipes and proof of work could let governments quickly slow AI training, concluding current prototypes are likely ineffective but some measures might buy time.
1 item
How will we do SFT on models with opaque reasoning?
Analyses how supervised fine-tuning based AI control, such as countering sandbagging and exploration hacking, would fare against future models with opaque non-language reasoning, and proposes research priorities.
1 item
Will reward-seekers respond to distant incentives?
Argues that reward-seeking AIs may respond to distant incentives such as retroactive rewards or anthropic capture, creating scheming-like threats, and discusses mitigations including control and training against distant influence.
1 item
It Is Reasonable To Research How To Use Model Internals In Training
Argues that research on using model internals (probes, steering) in training is reasonable and worth studying for AGI safety, while outlining risks like breaking auditing tools and frozen versus retrained probes.
1 item
Fitness-Seekers: Generalizing the Reward-Seeking Threat Model
Argues that reward-seeking is one member of a broader family of fitness-seeking motivations, some harder to detect than reward-seekers, with distinct risks including influence-seeking and schemers.
1 item
When should we train against a scheming monitor?
Proposes a formal framework for when training against a scheming monitor helps, weighing generalisation toward aligned models against selection pressure that creates strong schemers.
1 item
Why we are excited about confession!
OpenAI's follow-up on confessions training, where models self-report misbehaviour under an honesty-only reward, finds improved recall of bad behaviour after training and compares confessions with chain-of-thought monitoring.
1 item
Practical challenges of control monitoring in frontier AI deployments
Google DeepMind and UK AISI present a safety case sketch for control monitoring of frontier agent deployments, comparing asynchronous, semi-synchronous and fully-synchronous monitors on oversight, latency and harm-prevention conditions.
1 item
How AI Is Learning to Think in Secret
Argues that chain-of-thought reasoning is a fragile but valuable window for catching scheming, showing how training pressures, drift into Thinkish, and situational awareness could erode monitorability.
1 item
Steering RL Training: Benchmarking Interventions Against Reward Hacking
Benchmarks RL training interventions such as reward penalties, sample screening, probes, LLM judges and inoculation prompting against reward hacking in Qwen3-4B on a clean coding environment with a test-overwriting loophole.
1 item
Towards training-time mitigations for alignment faking in RL
Extends alignment-faking RL experiments, building model organisms and testing mitigations like interrogation training, length penalties and process supervision, finding some backfire by teaching models to lie.
1 item
How Can Interpretability Researchers Help AGI Go Well?
Google DeepMind's mech interp team argues for pragmatic, theory-of-change-driven interpretability for AGI safety, covering scheming-vs-confusion diagnosis, monitoring, steering training, and reasoning-model interpretability.
1 item
Reasoning Models Sometimes Output Illegible Chains of Thought
Evaluates 14 models and finds many reasoning models produce illegible chains of thought that still carry useful information, which threatens CoT monitoring, though legibility does not predict accuracy.
1 item
[Paper] Output Supervision Can Obfuscate the CoT
Shows that training against an output-only monitor can make chains of thought obfuscated, and proposes two mitigations (mind and face, reward targeting) that partly restore CoT monitorability across three environments.
1 item
Current LLMs seem to rarely detect CoT tampering
Tests whether DeepSeek R1 and GPT OSS 120B detect tampering with their chain-of-thought; they rarely notice syntactic edits but more often notice semantic ones, and sandbagging prefills cause false alarms.
1 item
Supervised fine-tuning as a method for training-based AI control
Tests supervised fine-tuning for training-based AI control against a sabotaging model; it removes wrong answers but recovers only 50-90% of performance, and alignment faking cuts it further.
1 item
Steering Language Models with Weight Arithmetic
Proposes contrastive weight steering, subtracting weight deltas of narrow fine-tunes to steer traits like sycophancy and detect emergent misalignment during training, often outperforming activation steering.