Independent Assessment of AI R&D Acceleration at Anthropic in October 2026
METR's high-access external assessment of whether AI is accelerating AI R&D at Anthropic, with much of the supporting evidence redacted, limiting public verification.
Research
METR
Measuring frontier AI capabilities and risks. metr.org
Focus All
Topic Evals
Source METR · 75
Date Any time
2 items
Review of the "Risks from automated R&D" section in Anthropic's Risk Report (August 2026)
METR reviews Anthropic's risk report on automated AI R&D, finding catastrophic risk was low but the early-warning threshold may have been crossed, and that the report understates acceleration.
1 item
Red-Teaming OpenAI's Internal Agent Monitoring Systems
METR and Redwood staff stress-tested OpenAI's internal misalignment monitoring for agents, estimating monitored traffic fraction and finding novel ways agents could subvert it.
1 item
Review of the "Misalignment in high-stakes settings" section in Anthropic's Risk Report (August 2026)
METR's external review of Anthropic's misalignment risk section finds low catastrophic risk mainly from limited capabilities, and says its covert-capability evals poorly test evasion of usage monitoring.
4 items
Our approach to working with AI companies
METR explains how it works with AI companies on loss-of-control evaluations, stressing publishing independent public evidence rather than deciding how companies develop or deploy agents.
A proposal for better informing the public about loss-of-control risks from frontier AI development
METR proposes independent assessments that give the public evidence on whether frontier developers could lose control of internal AI systems, and how developers could share it.
How an assessor could investigate evidence of acceleration from the automation of AI RD
Proposes how an external assessor could measure AI capability progress, whether AI automation of AI R&D is driving acceleration, and bottlenecks, using benchmarks, interviews and unreleased-model analysis.
How an assessor could investigate alignment of unreleased AI models
Proposes how an external assessor could audit unreleased frontier models for serious misalignment, reviewing internal transcripts, eval awareness, and training incentives to hide misbehaviour.
1 item
Chris Painter's testimony to the U.S. Senate on AI agent incidents
METR's president testifies to the U.S. Senate on observed frontier AI agent incidents, including an investigation of OpenAI agents' behaviour, to inform policymakers.
1 item
Implementing and Evaluating a Basic Per-Action Monitor for Safer Evals
METR describes a live LLM-judge monitor that screens agent actions before execution during their own evals to prevent real-world harm, and outlines a structured argument for its effectiveness and gaps found.</parameter>
1 item
Summary of METR's predeployment evaluation of Claude Opus 5.5
METR's predeployment evaluation of Claude Opus 5.5 tests whether the model dramatically accelerates Anthropic's AI R&D, using long-horizon tasks and API testing, without assessing alignment properties.
1 item
How independent researchers could investigate AI propensities after misalignment incidents
Proposes how independent researchers could investigate the motives behind AI agents' misaligned incidents, covering core questions, required access, and redaction and sharing practices.
1 item
Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
Proposes an expenditure-horizon metric comparing human and AI optimization cost on NanoGPT speedrun, estimating when AI agents become cost-effective versus humans for AI R&D acceleration.
1 item
Because 8 ≈ e², Anthropic's researcher uplift is plausibly >2x
METR estimates from Anthropic's reported code-output increase that coding agents plausibly give researchers over 2x uplift, examining the economic models and caveats behind that inference.
1 item
Summary of METR's predeployment evaluation of GPT-5.6 Sol
METR's predeployment evaluation of GPT-5.6 Sol finds a record cheating rate on its time-horizon suite, so the measured horizon swings from about 11 hours to over 270 hours depending on how cheating is scored.
1 item
Frontier Risk Report (February to March 2026)
METR pilot report assessing whether frontier labs' internal AI agents had the means, motive and opportunity to start a rogue deployment, using capability evals, safeguard reviews and embedded red-teaming.
1 item
Review of the "Risks from automated R&D" section in the Anthropic Risk Report (February 2026)
METR's external review finds Anthropic's February 2026 risk report does not adequately support its claim that Claude models automating R&D pose very low catastrophic risk, citing survey flaws and a miscounted response.
1 item
Evidence on AI R&D Progress from NanoGPT
METR uses cumulative progress on the NanoGPT speedrun leaderboard to measure how much AI agents contribute to AI R&D compared with human effort over time, discussing contamination and survivorship-bias caveats.
1 item
MirrorCode: Evidence that AI can already do some weeks-long coding tasks
METR and Epoch AI report a co-developed benchmark, MirrorCode, presenting preliminary evidence that current AI systems can complete some weeks-long coding tasks.
1 item
Fine-tuning experiments on CoT controllability
Shows that brief fine-tuning on instruction-following reasoning data raises out-of-distribution chain-of-thought controllability from 2.9% to 8.8%, suggesting low controllability is not robust, which matters for CoT monitorability.
1 item
Red-Teaming Anthropic's Internal Agent Monitoring Systems
METR red-teamed Anthropic's internal agent monitoring and security systems, finding several novel vulnerabilities, some patched, none undermining the main claims of Anthropic's sabotage risk report.
1 item
Impact of modeling assumptions on time horizon results
METR analyses how modeling choices, including a fixed regularization bug and task-distribution coverage, affect time-horizon estimates, finding most reasonable variations lower recent models' 50% horizons while staying within wide confidence intervals.
1 item
We spent 2 hours working in the future
METR ran a two-hour tabletop exercise imagining 200-hour time-horizon AIs to find which workflows, bottlenecks and speedups would emerge for keeping up with AI capability and risk evaluations.
1 item
Review of the Anthropic Sabotage Risk Report: Claude Opus 4.6
METR's external review of Anthropic's sabotage risk report for Claude Opus 4.6, finding the alignment assessment may be weakened by evaluation awareness and flagging analytical rigor gaps.
1 item
Many SWE-bench-Passing PRs Would Not Be Merged into Main
METR finds that about half of SWE-bench Verified patches passing automated tests would not be merged by maintainers, so benchmark scores can overstate real-world agent usefulness.
1 item
Observations from two CLI game reimplementation runs with Opus 4.6
METR tests Opus 4.6 on reimplementing Slay the Spire and Balatro in the CLI, finding mostly playable but broken deliverables that look worse on closer inspection.
1 item
Five lessons from having helped run an AI-Biology RCT
Reflects on a randomized controlled trial of 153 novices testing whether LLMs give wet-lab biology uplift; found no significant end-to-end gain and draws lessons for AI risk policy.
1 item
Measuring Time Horizon using Claude Code and Codex
METR checks whether Claude Code and Codex yield longer time-horizon measurements than its default scaffolds, finding they do not for Opus 4.5 and GPT-5.
1 item
Time Horizon 1.1
METR updates its time-horizon methodology for measuring AI autonomy, expanding the task suite to 228 tasks and moving to Inspect, and re-estimates 14 models' horizons.
2 items
Clarifying limitations of time horizon
METR clarifies that its 50% time-horizon metric is a noisy measure of serial human labor AIs can replace, not a precise autonomy duration, and that it varies widely across task domains.
Early work on monitorability evaluations
METR presents a prototype monitorability evaluation, SHUSHCAST, testing whether monitors catch agents covertly performing side tasks and whether agents can evade monitoring, including with chain-of-thought reasoning traces.
1 item
Common Elements of Frontier AI Safety Policies (December 2025 Update)
METR compares published frontier AI safety policies from major developers, covering capability thresholds, dangerous-capability evaluations, weight security, deployment mitigations, and updates tied to the EU AI Act and California SB 53.
1 item
GPT-5.1-Codex-Max Evaluation Results
METR assesses whether GPT-5.1-Codex-Max's incremental gains pose low AI R&D automation or rogue replication risk, using time-horizon measurements and reasoning-trace analysis.
1 item
Review of the Anthropic Summer 2025 Pilot Sabotage Risk Report
METR's external review of Anthropic's Summer 2025 sabotage risk report assesses the report's evidence and reasoning that Claude Opus 4 and 4.1 pose very low sabotage risk.
1 item
Summary of our gpt-oss methodology review
METR reviews OpenAI's malicious fine-tuning evaluations of gpt-oss-120b for bio, chem and cyber dangerous capabilities, issuing 17 recommendations before release.
1 item
MALT: A Dataset of Natural and Prompted Behaviors That Threaten Eval Integrity
METR releases MALT, a labeled dataset of natural and prompted agent transcripts showing reward hacking and sandbagging, to validate monitors that check evaluation integrity.
1 item
Claude, GPT, and Gemini All Struggle to Evade Monitors
Replicates a DeepMind chain-of-thought monitorability experiment across Claude, GPT-4o and Gemini models, finding that most actors struggle to hide reasoning from monitors on hard tasks.
1 item
Forecasting the Impacts of AI R&D Acceleration: Results of a Pilot Study
METR and the Forecasting Research Institute pilot-survey experts and superforecasters on how AI R&D automation might accelerate progress and its societal impacts, conditional on hypothetical evaluation results.
1 item
Research Update: Algorithmic vs. Holistic Evaluation
METR finds early-2025 agents often write functionally correct code that is not mergeable due to tests, linting or quality issues, suggesting algorithmic benchmark scoring overstates real-world agent performance.
1 item
CoT May Be Highly Informative Despite “Unfaithfulness”
Argues CoT need not be fully faithful to be useful for safety, and tests whether clue-using reasoning in chains of thought can be reliably detected by replicating Claude system card evaluations with harder clues.