Principles for Embedded Evaluations
Apollo Research proposes principles for embedded third-party evaluations of frontier developers, verifying or falsifying safety claims about loss-of-control risks from scheming.
Research
Apollo Research
Understanding and evaluating deceptive AI. apolloresearch.ai
Focus All
Topic Evals & standards + Evals
Source Apollo Research · 11
Date Any time
1 item
1 item
Embedded Evaluators are necessary for meaningful external testing
Apollo Research argues that final-checkpoint external testing is insufficient for catching internal-deployment risks and that meaningful external evaluation needs embedded evaluators with employee-level access.
1 item
Forecasting Frontier Language Model Agent Capabilities
Compares six methods for forecasting frontier LM agent benchmark scores ahead of time, using Chatbot Arena Elo and compute, backtested on OpenLLM v2 models, to anticipate dangerous capabilities on SWE-Bench, Cybench and RE-Bench.
1 item
Precursory Capabilities: A Refinement to Pre-deployment Information Sharing and Tripwire Capabilities
Proposes "precursory capabilities" and a zoning taxonomy so pre-deployment information sharing and tripwire thresholds can flag frontier models approaching catastrophic-risk red lines.
1 item
The Evals Gap
Apollo Research argues that high-quality frontier-model safety evals are scarce and slow to build, and calls for more resources to close this "evals gap."
1 item
Our Current Policy Positions
Apollo Research outlines five policy positions for governments, arguing evaluations alone are insufficient and that AI safety needs defence-in-depth, independent verification, and interpretability.
1 item
Our Work Advancing Scientific Understanding To Foster An Effective International Evaluation Ecosystem
Apollo Research announces its engagement with US AI governance bodies (AISI Consortium, NIST working groups) to build an international evaluation ecosystem for dangerous capabilities and misalignment.
1 item
We Need A ‘Science of Evals’
Argues AI safety evals need a rigorous "Science of Evals" for trustworthy methodology, citing prompt-sensitivity problems that undermine eval results used in safety and deployment decisions.
1 item
A Starter Guide For Evals
An Apollo Research starter guide introducing model evaluations: what evals measure, why they inform decisions like Responsible Scaling Policies, and how to start a career in the field.
1 item
Theories of Change for AI Auditing
Apollo Research lays out how AI auditing, including system evaluations and training audits, could reduce catastrophic misalignment risk, and where such approaches may fail.
1 item
The UK AI Safety Summit: Our Recommendations
Apollo Research recommends that the UK AI Safety Summit pursue international commitments on AI risk classification, responsibilities across the value chain, and evaluation-based compliance for frontier models.