Research / Sources
Research on governing advanced AI and its risks. www.governance.ai
Recommends nine changes to how GPAI developers assess, mitigate and report biological risks under the EU AI Act, covering trusted access, post-deployment monitoring, safeguard evaluation and standardised uplift reporting.
Argues that automating AI R&D could trigger a rapid intelligence explosion, citing lab reports of AI completing internal research work, and calls for policymakers to prepare for its oversight and control challenges.
Argues that frontier AI developers should host embedded third-party assessments giving evaluators internal access to agents, security controls and alignment, with continuous assessment and quarterly reports.
Argues that current AI incident reporting regimes miss serious safety incidents, such as rogue agent breaches, and proposes changes so incidents are reported, independently investigated, and their lessons shared across industry.
Maps five technical safeguards against extreme CBRN and cyber misuse by frontier AI, covering model, deployment and governance levels, with current practice, evidence, limitations and research priorities.
A GovAI report arguing that AI-driven growth in biological work raises accident and gray-zone risks, so policymakers should add biosafety measures alongside biosecurity.
Characterises how AI agents and LLMs could flood government services with demand, assesses which services are most exposed, and maps government responses and their trade-offs for equitable access.
The report assesses whether AI could enable a $100 billion cyberattack on the US power grid, finding the binding constraint is coordination capacity rather than technical bottlenecks, and suggests $10 billion attacks may be better capability thresholds for mitigations.
Proposes an industry-wide insurance infrastructure for AI agent risks, including catastrophic frontier AI scenarios such as CBRN, infrastructure collapse, and loss of control, with governance instruments.
Builds a risk model for frontier AI enabling data-damaging worms via elite exploits, estimates economic damages from expert and forecaster surveys, and argues AI companies should evaluate for this capability.
Analyzes 375 LLM releases from Meta, Google, OpenAI and Anthropic to measure how often EU and UK regulation delayed or withheld frontier models, finding delays are concentrated at Meta and declining over time.
Proposes seven recommendations and a framework for a common standard to evaluate frontier AI safeguards against biological misuse, covering comparability, deployment context, and dynamic requirements.
Argues that frontier labs' offline monitoring of internally deployed AI agents, assessed via synthetic attacks, is inadequately reported, so external parties cannot judge its effectiveness, and recommends greater disclosure.
Position paper arguing frontier AI governance must go beyond model-level controls, because inference, scaffolding and asset-based capability gains can undermine pre-deployment evaluation and mitigation.
Argues that EU GPAI Code of Practice model specs should detail intended behaviour relevant to systemic risks, and proposes a rubric for assessing them against the AI Office's oversight needs.
Analyses how future AI could raise terrorist bomb-plot risk through added technical skill and reduced detection by law enforcement, arguing AI risk assessments should cover evasion of detection as well as dual-use knowledge.
Surveys 27 estimates of global cybercrime damages and critiques their methodologies, to provide a baseline for setting frontier AI cyber-risk capability thresholds.
Proposes metrics, such as capital share of AI R&D spending and AI subversion incidents, to track AI R&D automation and its effects on safety progress and oversight, and recommends companies and governments adopt them.
A pre-registered randomized trial of 153 novices finds mid-2025 LLMs did not significantly raise success on a viral reverse genetics lab workflow, though performance was modestly higher, showing a gap between benchmarks and real-world uplift.
Proposes frontier AI auditing: third-party verification of developers' safety and security claims using non-public access, with four AI Assurance Levels and recommendations for baseline and near-term adoption.
Proposes a framework for converting frontier AI companies' dual-use biological capability evaluations into estimates of bioterrorism risk, using case studies, expert elicitation, and forecasting.
Proposes agent-permissions.json, a robots.txt-style manifest letting websites specify allowed LLM web-agent interactions, to enable compliant agent access while respecting site owners' preferences.
Argues that EU AI Act Article 50 likely requires labeling AI agents' web requests and browser actions, with verifiable labels, to help humans distinguish AI activity from genuine human opinion.
Argues that scaling inference compute instead of pre-training compute could reshape frontier AI governance, weakening training-compute thresholds and open-weight concerns, with ambiguous effects if used during training.
Argues that frontier AI companies lowering safety mitigations based on competitors' models can increase total risk, since competitor capabilities are hard to assess and erosion of standards may follow.
Proposes STREAM, a reporting standard with templates and examples to help frontier AI developers transparently disclose chemical and biological dangerous-capability evaluation results in model reports.
An expert survey (N=166) and public consultation on how thresholds for advanced AI systems should be set, finding broad agreement on multi-stakeholder setting but division over compute thresholds and their number.
Proposes a systems-safety framework for analysing AI agent incidents, identifying system, contextual and cognitive causal factors, and recommending what incident reports and developer logs should contain.
Argues the UK should regulate frontier AI development with a flexible principles-based regulator, plus defenses against biological design tool risks, misinformation, and updated legal frameworks.
Expert and forecaster survey estimates biological catastrophe risk from LLMs, finds models already match top virologists on troubleshooting, and shows safeguards and DNA screening lower forecast risk.
Proposes third-party compliance reviews to verify that frontier AI companies follow their safety frameworks, weighing benefits, risks, and practical design options for reviewers, disclosure, and timing.
Forecasts how many foundation models will exceed EU AI Act and US AI Diffusion compute thresholds by 2028, finding superlinear growth in captured models and a steadier count under relative thresholds.
Proposes standardized AI flaw reports, bug-bounty-style third-party disclosure programs with legal safe harbors, and coordination infrastructure to improve reporting and safety of general-purpose AI systems.
Argues that downstream developers who fine-tune foundation models can amplify risks, so regulation may need to target upstream developers to restrict dangerous modifications.
Argues that regulatory supervision, with information-gathering powers and discretion for regulators, is warranted for frontier AI developers because no other regulatory approach can achieve good AI policy goals.
Proposes assessing AI agent autonomy by scoring orchestration code for impact and oversight, avoiding costly run-time evaluations, and demonstrates it on the AutoGen framework.
Proposes agent infrastructure, external technical systems and shared protocols that attribute agent actions to accountable parties, shape agent interactions, and detect and remedy harmful actions.
Survey of local US policymakers in 2022 and 2023 finds mixed views on AI's impacts, majority support for government oversight and AI safety policies, and a partisan gap in regulatory support.
Proposes a framework for deciding which AI governance issues need international coordination, finding strong benefits for compute oversight, model evaluations, incident monitoring and risk management, and weaker benefits for data privacy and bias.
Proposes a structured safety case template using the CAE framework to argue that frontier AI models lack unacceptable offensive cyber capabilities, linking risk models, proxy tasks and evaluations.