Research / Sources
Research on governing advanced AI and its risks. www.governance.ai
Recommends nine changes to how GPAI developers assess, mitigate and report biological risks under the EU AI Act, covering trusted access, post-deployment monitoring, safeguard evaluation and standardised uplift reporting.
Argues that frontier AI developers should host embedded third-party assessments giving evaluators internal access to agents, security controls and alignment, with continuous assessment and quarterly reports.
Maps five technical safeguards against extreme CBRN and cyber misuse by frontier AI, covering model, deployment and governance levels, with current practice, evidence, limitations and research priorities.
The report assesses whether AI could enable a $100 billion cyberattack on the US power grid, finding the binding constraint is coordination capacity rather than technical bottlenecks, and suggests $10 billion attacks may be better capability thresholds for mitigations.
Builds a risk model for frontier AI enabling data-damaging worms via elite exploits, estimates economic damages from expert and forecaster surveys, and argues AI companies should evaluate for this capability.
Argues that frontier labs' offline monitoring of internally deployed AI agents, assessed via synthetic attacks, is inadequately reported, so external parties cannot judge its effectiveness, and recommends greater disclosure.
Analyses how future AI could raise terrorist bomb-plot risk through added technical skill and reduced detection by law enforcement, arguing AI risk assessments should cover evasion of detection as well as dual-use knowledge.
A pre-registered randomized trial of 153 novices finds mid-2025 LLMs did not significantly raise success on a viral reverse genetics lab workflow, though performance was modestly higher, showing a gap between benchmarks and real-world uplift.
Proposes a framework for converting frontier AI companies' dual-use biological capability evaluations into estimates of bioterrorism risk, using case studies, expert elicitation, and forecasting.
Proposes STREAM, a reporting standard with templates and examples to help frontier AI developers transparently disclose chemical and biological dangerous-capability evaluation results in model reports.
Expert and forecaster survey estimates biological catastrophe risk from LLMs, finds models already match top virologists on troubleshooting, and shows safeguards and DNA screening lower forecast risk.
Proposes assessing AI agent autonomy by scoring orchestration code for impact and oversight, avoiding costly run-time evaluations, and demonstrates it on the AutoGen framework.
Proposes a structured safety case template using the CAE framework to argue that frontier AI models lack unacceptable offensive cyber capabilities, linking risk models, proxy tasks and evaluations.
GovAI's response to NIST's RFI recommends proportionate risk management, safety policies, evaluations, red-teaming and post-deployment monitoring for dual-use foundation models.
Proposes a five-step scheme in which frontier developers pause R&D when dangerous-capability evaluations fail, and compares voluntary, collective, auditor-based, and legally required versions.
An opinion piece by Anderljung and Scharre arguing that society must prepare for powerful AI, using a case where an AI system was induced to plan synthesis of chemical weapons to motivate frontier AI governance.
GovAI's submission to NTIA argues foundation models need independent audits against dangerous capabilities, alignment and truthfulness, and recommends government actions to build an auditing ecosystem.
Argues that dangerous-capability and alignment evaluations are essential for identifying extreme risks from general-purpose AI and informing policymakers and decisions on training, deployment and security.
Tests whether LLMs can assist spear phishing, generating emails for 600+ UK MPs, bypassing safeguards via prompt engineering, and proposes structured access and LLM-based defenses.