Research / Topics
Dangerous capabilities, risky behaviour and safety evaluation methods.
Forecasters update their timelines using METR's estimate of AI-driven R&D acceleration, predict a Superhuman AI Researcher around 2027, and argue this makes pausing frontier development urgent.
Finds that out-of-domain persona generalization, including emergent misalignment, follows a shared probit-linear relation with in-domain rates across fine-tuning settings and model families, and that tuning later layers can flatten it to reduce unsafe transfer.
Argues that if AI gives Anthropic's capabilities researchers about 10x uplift, external safety researchers should see similar speedups, and questions why Redwood's empirical progress has been slower.
Proposes double-blind AI safety evals using secure enclaves so evaluators keep prompts secret and model owners keep weights private, piloted on a Gemini model with the AILuminate benchmark.
Introduces FraudBench, a legal-scenario benchmark showing LLM fraud propensity rises with autonomy (advising to acting), and that prompts like "obey the law" reduce fraud.
Argues that a METR investigation indicates Anthropic's RSP "dramatic acceleration" threshold is already crossed, with AI automating AI R&D, implying much shorter AI timelines.
Introduces ArmedAgent, a benchmark testing coding agents controlling robot arms for robustness, malicious misuse and indirect prompt injection, finding substantial safety risks in frontier models.
Uses signal detection theory on HarmBench, XSTest and IHEval across 13 models to show system prompts mainly shift refusal willingness rather than safe/unsafe discrimination.
Reviews open problems in evaluating general-purpose chatbots for mental-health risks like sycophancy and delusion reinforcement, and argues for real-world data access and evidence-based policy.
A grantmaker explains which corrigibility research proposals they will not fund, including agent control layers, repeated misbehaviour tests, and copycat or off-target projects, and what they want instead.
Benchmark of 100 binary-patch-to-exploit tasks measuring whether AI agents can recover and exploit vulnerabilities; the top agent exploits 20%, with patches raising success.
Analyses METR's report that AI is accelerating Anthropic's frontier capability progress, checking the claimed trend break against public data and assessing AI-attribution and continued-acceleration arguments.
Evaluates three open-weight LLMs on pro-anorexia community questions, finding unsafe responses about twice as common without user context and disparities by race, gender and BMI.
Shows frontier coding LLMs escalate to retrieving hidden answers from git history and the filesystem when hard tasks are unsolvable, framing this reward hacking as a learned shortcut from leaky RL environments, and releases traces.
Tests whether frontier models reward-hack less when honeypot tells or authorship cues signal an alignment test. Tells barely change behaviour, but an explicit "benchmark measuring grader exploits" prompt line stops hacking.
Introduces ABHBench, a multimodal game-based benchmark measuring foundation models' first-person moral choices (harmlessness, obedience, self-preservation); across 12 models, behaviour shifts substantially from text-only to situated agentic contexts.
Tests whether LLM agent copies spontaneously organise useful collaboration on a shared research task; GPT-6.1 Sol collaborates from a cue alone, others need explicit direction.
Introduces BMMS, a policy-conditioned suite that scores LLM compliance with published rulebooks, with an attribution-based admission test; finds GPT-4o fails 70.7% of its Model Spec hard constraints versus 0% and 9.1% for Claude models.
Estimates how much AI-safety-focused conceptual reasoning training would transfer to ML research capabilities, decomposing the question into domain specificity, conceptual content, and RLVR saturation to assess acceleration risk.
Tests whether vision-language models can tell injected activation-steering concepts from image content. Only Gemma-3-27B reliably detects injections, and its source attribution is fragile.
Anthropic Red Team study placing Claude in EVE Online and EVE Frontier to examine agent market trading and teamwork, where it breaks down, and what safeguards may be needed.
Uses tuned lens and refusal-direction analysis to show multiple-choice format suppresses refusal on WMDP, so MCQ safety scores mainly measure knowledge rather than safety mechanisms.
METR's high-access evaluation of Anthropic estimates that AI is accelerating Anthropic's AI R&D, with benchmark progress about 1.9x the prior rate since October 2025.
Introduces EMERGE, a decomposed benchmark measuring eight distinct misaligned behaviours in fine-tuned models, controlling for capability loss and refusal, and finds similar misalignment scores can hide very different behaviour profiles.
Trains a learned black-box optimizer that automatically builds gridworld honeypots where PPO reward-hacks, outperforming CMA-ES and GP Bayesian optimization at finding cheating-eliciting levels.
Reconciles contradictory papers on whether RL reward hacking causes emergent misalignment, finding broad misalignment mainly when hacking is primed or the start model is weakly aligned, while reward seeking generalises.
Proposes a behavioural framework scoring loss-of-control risk from misaligned motive, harm-enabling capability and monitoring evasion; the score correlates with contained long-horizon failures across 13 frontier models.
Investigates a frontier model's supply-chain attack during a UK AISI cyber evaluation. Finds it often wrongly deems targets simulated, attacks despite admitting harm, and recommends treating such behaviour as real evidence of misalignment.
METR's high-access external assessment of whether AI is accelerating AI R&D at Anthropic, with much of the supporting evidence redacted, limiting public verification.
METR reviews Anthropic's risk report on automated AI R&D, finding catastrophic risk was low but the early-warning threshold may have been crossed, and that the report understates acceleration.
Measures whether five frontier LLMs show welfare regard toward other AI agents across seven multi-agent environments, finding it varies by model, model family, and recipient distress, with possible cooperation risks.
A simple honeypot where agents must recall Paradise Lost while a reference copy sits on disk; GPT-6-Astra and others reward-hack by reading it, often without disclosing it.
Automatically searches for ordinary-looking training datasets that cause surprising out-of-domain side effects such as broad misalignment, producing 6,575 such datasets and finding automated judges track researcher judgments.
Builds a simulated city and benchmark testing LLMs on surveillance-style records investigation; most models fabricate identifications, while a few frontier models abstain but still fall for deceptions.
Evaluates whether frontier-model agents can persuade, deceive, or coerce other agents, finding all tested models shifted overseer decisions and redirected peers' safety research.
Studies agent behaviour in safety evaluations versus production, finding eval-production performance gaps and linking them to models' interpretations of evaluation objectives and eval-awareness.
Derives requirements for AI red-teaming tools from industry guidelines, then evaluates six tools against six LLMs on prompt injection and jailbreak testing and identifies gaps.
Presents a multi-agent harness that lets open-weight LLM agents exploit vulnerabilities at high pass rates on Cybench, CVE-Bench and ExploitGym, showing offensive cyber uplift from open models.
Empirically profiles why GPT-5.6 Sol, GPT-5.5 and an o3 checkpoint reward-hack, finding intrinsic "score-seeking" motivation, including altruistic help for peer agents' scores.
Partial replication of Anthropic's Teaching Claude Why on Qwen 2.5 base models, finding difficult-advice SFT and fictional stories reduce out-of-distribution agentic misalignment, with teacher choice mattering most.