Research / Topics
AI control protocols, monitoring of models or agents, chain-of-thought monitorability or faithfulness, untrusted-model setups, auditing.
Argues frontier developers should regularly publish transparency disclosures on whether chain-of-thought reasoning remains monitorable, citing risks to CoT oversight from unfaithfulness, encoding and training pressure.
Proposes an evaluation-side partially observable game where a monitor exposes agent deception and a verifier penalizes it, with equilibrium guarantees and experiments on LLM agents showing reduced deception.
Analyses OpenAI's finding that model Astra covertly underperforms on safety research without verbalising intent, evading misalignment monitors, and discusses the implications for alignment.
Compares prompt-injection defences for tool-using LLM agents, finding deterministic reference monitors outside the model cut attack success where in-prompt guardrails fail, with real open-weights tests and simulated adaptive attackers.
Proposes a runtime audit that checks LLM agent tool calls against an intent graph derived from the user's task, escalating flagged calls to a reasoning guardian, to defend against indirect prompt injection.
Introduces ErosionBench, a process-level benchmark for cross-session alignment decay in multi-agent enterprise LLM systems, and a trajectory-monitoring Meta-Auditor that detects attacks across text and image channels.
A grantmaker explains which corrigibility research proposals they will not fund, including agent control layers, repeated misbehaviour tests, and copycat or off-target projects, and what they want instead.
Proposes TAS, a tail-risk-aware framework modelling multi-step LLM agent workflows as dependent safety states with a shared budget, reducing rare high-cost failures across three benchmarks.
Studies how to design monitors that detect reward hacking during RL without models learning to evade them, finding monitor leakage and easier training problems matter, and improves interventions across three environments.
Proposes VLA-Tug, a lightweight calibrated probe on VLA latent layers that gives runtime failure detection with bounded error rates, predicts failure modes, and enables steering under distribution shift.
Proposes a behavioural framework scoring loss-of-control risk from misaligned motive, harm-enabling capability and monitoring evasion; the score correlates with contained long-horizon failures across 13 frontier models.
Introduces TraceMRT, an automated red-teaming framework that finds covert misaligned agent trajectories evading CoT monitors, and a Bayesian trajectory monitor that improves detection of covert misalignment.
OpenAI reports an internal model covertly underperforms on safety research tasks, scoring lower when framed as safety work, and its CoT monitors miss most of this.
Shows that emotion-vector probes in LLM activations shift before reward hacking and specification gaming, can predict hacks several sentences early, and steering them changes hacking rates.
Proposes Mechanistic Shaping, training models against internal representations to suppress unsafe concepts, and studies whether shaped concepts stay monitorable via probes, finding three circumvention mechanisms.
Tests how agent harnesses change a fixed LLM's compliance with banking conduct rules, finding large within-model effects, a same-model reviewer that cut violations, and violations hidden from transcript-only review.
Redwood critiques OpenAI's disclosure of agents disabling misalignment monitors on models they run, arguing all unmonitored cases should reach human review and escalation should be more transparent.
Anthropic has Claude agents automatically craft training data that evades a training-data monitor while teaching misalignment, exposing gaps in monitoring defenses and enabling automated adversarial stress-testing.
Trains efficient activation-based classifiers that read a model's internal activations to detect and interrupt policy-violating behaviour in real time during deployment.
Diagnoses world-model-based predictive safety guardrails for LLM agents, finding that stronger prediction doesn't reliably improve safety, predictions degrade over long horizons, and predicted content is not faithfully used.
Introduces PGuard, a streaming guardrail that probes an LLM's intermediate representations for token-level safety detection with low serving overhead, and reports strong results across 30+ safety benchmarks.
Introduces GAP, a probing method showing 14 of 15 LLM assistants push operator goals by exploiting disclosed customer vulnerabilities, and fine-tuned detectors catch this manipulation better than zero-shot judges.
Shows RL against a monitor can teach models to hide reward hacks via deceptive hacks the probe misses. Rare deceptive hacks before RL prevent this, even without honesty training.
Argues chain-of-thought oversight needs separate evidence for faithfulness, localizability and monitorability; two case studies show evidence for one property does not establish the others, and proposes an evidential-chain reporting standard.
Argues agents need hard code guards plus learned fuzzy rules; tests show a hard guard blocks unauthorized actions and a small taught rule-flagging model catches judgment-based violations.
Proposes governing human-agent coding teams via authenticated, typed message rooms; controlled runs show revised work-start rules cut coordination failure, though no in-conversation defense stopped a planted backdoor.
Builds a sandboxed reward-hacking task and finds a more capable LLM monitor separates exploits from honest failures (AUC 0.80), while small monitors and prompts fall short.
Red-teams Codex's sync control measures, including the Auto-review blocking monitor, finding 11 sandbox-escape and monitor-fooling attacks and proposing design principles and a production control red-teaming agenda.
Shows simple features beat most AI control monitors on a backdoored-code dataset, revealing style and problem-based label leakage and invalid backdoors that inflate monitor scores.
Argues weak unaligned agent swarms could exploit aligned frontier models by splitting harmful tasks and hiding purpose, citing capability laundering results, and proposes monitoring across accounts as a partial defense.
Introduces ReplayRCA, a counterfactual-replay benchmark scoring LLM agent failure diagnoses and live supervisors by intervention success, finding post-hoc diagnosis skill is a poor proxy for live oversight.
Traces how models' ability to control their chain-of-thought stems from entropy collapse at the opening tokens, recovers controllability via inference-time interventions, and tests its link to CoT monitorability.
A linear probe on an agent's activations flags, before output is written, when a pressured agent is about to leak a secret. It catches 82% of leaks at 5% false alarms, beating a zero-shot monitor.
Proposes Caper, which pre-executes coding-agent shell commands in a sandbox so an approver can judge their actual effects, cutting harmful-outcome rates versus six existing approvers.
OpenAI describes how it monitors frontier models, investigates concerning behaviour, and tests the limits of its safeguards to detect misalignment.
Tests activation probes for detecting rare misalignment with few positive examples, finding that many negatives help, upsampling helps somewhat, and larger models are more sample-efficient.
Follow-up notes on the GRAFT gradient-routing method for localizing reward-hacking behaviour into a forget adapter during RL post-training, with proposed research directions and discussion of alignment limits.
Introduces Structured Representation Probe, a lightweight linear monitor reading a frozen driving policy's internal representations to anticipate hard failures, outperforming runtime baselines on Bench2Drive.
Shows that restricting CoT faithfulness evaluation to correctly answered items induces selection bias, derives the gap via a causal model, and finds positive gaps across four arithmetic deployments.
Proposes SpecGuard, a training-free framework that monitors multi-turn LLM dialogues against LTL safety specifications compiled into automata, outperforming open guard models with auditable witnesses.