Control monitors need alignment assessments too
Argues that AI control monitors can fail through misalignment, not just weak capability, so their alignment needs assessing via arguments and evals, since evals alone are insufficient.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Control + Evals
Source LessWrong · 23
Date Any time
1 item
2 items
GLM-5.1 Has ~Astra Level CoT Control
Evaluates chain-of-thought controllability across open models, finding GLM-5.1 highly controllable but poor at evading CoT monitors for sandbagging, hacking and gaslighting tasks.
More Capable Models Write Better Probe Evasion Prompts
Tests whether LLMs can write prompts that evade white-box probe monitors while completing tasks, finding stronger models with more feedback evade better, a concerning early indicator of in-context probe evasion.
1 item
Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming
Introduces Screwtape, a pipeline that iteratively edits real agent transcripts to create realistic attacks evading action-only monitors, finding six attacks that also evade a frontier developer's held-out monitor after human review.
1 item
Red vs Blue, but for Evals
Proposes framing AI evaluations as a red-team versus blue-team game in which a scheming model subverts evals via evaluation awareness, exploration hacking, or sandbagging, and analyses robustness versus richness of alignment and capability evals.
1 item
Scheming Evals Mislead in Both Directions
Shows that scheming evaluation detectors can mislead in both directions: a covert-action detector missed Claude Opus's transparent responses to harm, and a buggy scorer produced a false scheming signal for o1, arguing for hand-validating transcripts and measuring the response channel.
1 item
Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models
Measures how long a task frontier models can solve without chain of thought, finding no-CoT horizons doubling yearly, which matters for CoT monitoring and hidden scheming.
1 item
Retrying vs Resampling in AI Control
Revisits resampling and compares retrying protocols for AI control in BashArena, finding retrying with monitor feedback is exploitable and resampling aggregation choices matter for safety.
1 item
Introducing LinuxArena
Introduces LinuxArena, 20 realistic software engineering environments with side tasks and sabotage trajectories for evaluating AI sabotage capabilities, monitors, and control protocols.
1 item
Refactor Arena: A Control Setting for Software Engineering
Introduces Refactor Arena, a configurable control setting where untrusted agents refactoring codebases may insert delayed vulnerabilities, with deterministic scorers and initial evals validating the setup.
1 item
Attack Selection In Agentic AI Control Evals Can Decrease Safety
Shows that attack selection (start and stop policies) in agentic BashArena control evals substantially lowers safety against a trusted monitor, so fixed-attack-policy evaluations may overestimate protocol robustness.
1 item
Preliminary Explorations on Latent Side Task Uplift
Experiments test whether Claude Opus 4.5 and other models can solve arithmetic side tasks latently using filler tokens or phrase repetition in an AI-control setup, finding modest uplift mainly for Opus 4.5.
3 items
Reasoning Models Struggle to Control Their Chains of Thought
Introduces CoT-Control, a suite measuring whether reasoning models can control their chain-of-thought, finding low controllability that suggests CoT monitoring remains viable for now.
Salient Directions in AI Control
Lays out the structure of AI Control evaluations, covering monitor information channels, monitor capability, environments and side tasks, and proposes research workstreams for measuring control outcomes.
Investigating Self-Fulfilling Misalignment and Collusion in AI Control
Fine-tuning GPT-4.1 on AI control literature raised blackmail rates in the Agentic Misalignment scenario, while monitor collusion was absent except for a brittle, eval-aware Grok 4 result.
1 item
Monitoring benchmark for AI control
Introduces MonitoringBench, a dataset of human-seeded and refined attack trajectories for testing AI control monitors, finding that only Opus-4.5 reliably catches strong attacks and that monitors often fail to calibrate suspiciousness.
1 item
Schelling Coordination in LLMs: A Review
A literature review finding that LLMs coordinate on simple Schelling tasks but only weakly on covert channels and control subversion, arguing risk lies in coordination tied to strategic goals and threatens untrusted monitoring.
1 item
Methodological considerations in making malign initializations for control research
Methodological analysis of how red teams create malign model initializations for AI control evaluations, surveying creation methods and evaluating which blue-team mitigations and monitors defeat them.
1 item
BashArena: A Control Setting for Highly Privileged AI Agents
Introduces BashArena, a high-stakes AI control setting with 637 privileged sysadmin main tasks and four side tasks, and discusses its design lessons, strengths and weaknesses for evaluating control protocols.
1 item
Agentic Monitoring for AI Control
Investigates giving trusted monitors agentic tool use (blackbox and freeform code execution) to detect backdoors in AI control protocols, finding weaker models struggle with tools and the APPS setting is too easy to be informative.
1 item
Prompt optimization can enable AI control research
Uses DSPy prompt optimization, including GEPA, to automatically build stronger suspicion monitors for AI control protocols, raising safety against backdoored APPS code at low audit budgets.
1 item
AI Control Methods Literature Review
A literature review of AI control methods covering monitoring, control evaluations, guaranteed safety, and capability control for potentially misaligned AI systems.
1 item
Untrusted monitoring insights from watching ChatGPT play coordination games
Tests whether ChatGPT models can covertly coordinate via code signals in an untrusted-monitoring game, finding subtle signals evade detection and undermining synthetic input auditing in AI Control.