Research / Sources
Research on governing advanced AI and its risks. www.governance.ai
Argues that automating AI R&D could trigger a rapid intelligence explosion, citing lab reports of AI completing internal research work, and calls for policymakers to prepare for its oversight and control challenges.
Characterises how AI agents and LLMs could flood government services with demand, assesses which services are most exposed, and maps government responses and their trade-offs for equitable access.
The report assesses whether AI could enable a $100 billion cyberattack on the US power grid, finding the binding constraint is coordination capacity rather than technical bottlenecks, and suggests $10 billion attacks may be better capability thresholds for mitigations.
Builds a risk model for frontier AI enabling data-damaging worms via elite exploits, estimates economic damages from expert and forecaster surveys, and argues AI companies should evaluate for this capability.
Analyses how future AI could raise terrorist bomb-plot risk through added technical skill and reduced detection by law enforcement, arguing AI risk assessments should cover evasion of detection as well as dual-use knowledge.
Surveys 27 estimates of global cybercrime damages and critiques their methodologies, to provide a baseline for setting frontier AI cyber-risk capability thresholds.
Proposes metrics, such as capital share of AI R&D spending and AI subversion incidents, to track AI R&D automation and its effects on safety progress and oversight, and recommends companies and governments adopt them.
Proposes a framework for converting frontier AI companies' dual-use biological capability evaluations into estimates of bioterrorism risk, using case studies, expert elicitation, and forecasting.
Argues that scaling inference compute instead of pre-training compute could reshape frontier AI governance, weakening training-compute thresholds and open-weight concerns, with ambiguous effects if used during training.
Argues that frontier AI companies lowering safety mitigations based on competitors' models can increase total risk, since competitor capabilities are hard to assess and erosion of standards may follow.
Expert and forecaster survey estimates biological catastrophe risk from LLMs, finds models already match top virologists on troubleshooting, and shows safeguards and DNA screening lower forecast risk.
Forecasts how many foundation models will exceed EU AI Act and US AI Diffusion compute thresholds by 2028, finding superlinear growth in captured models and a steadier count under relative thresholds.
Argues that rising compute efficiency lets more actors train models to a given performance, so large compute investors keep a lead, and governments must prepare for widely available dangerous capabilities.
An opinion piece by Anderljung and Scharre arguing that society must prepare for powerful AI, using a case where an AI system was induced to plan synthesis of chemical weapons to motivate frontier AI governance.
Presents an interactive web app for a formal safety-performance tradeoff model of AI competition, examining why safety breakthroughs may not yield safer deployed systems.
A GovAI case study drawing lessons from atomic bomb development for governing transformative technologies safely, framed as a reference for future AI-era strategic risks.
A strategic model of AI-style technology races finds that laggards may cut corners on risky deployment when adversarial, and that safety improvements can be offset by risk compensation.
A game-theoretic model of AI development races shows how decisiveness and information disclosure affect disaster risk, with public capability information sometimes improving welfare.
Historical analysis of the 1944-1951 Baruch Plan debates on international control of nuclear technology, drawing lessons for governing existentially dangerous technologies such as AI.
Dafoe argues that AI governance is a neglected, high-impact cause area and outlines a longtermist theory of impact, while also preparing for conventional AI development scenarios.
Draws on the history of international regimes to weigh the trade-offs of centralising AI governance, concluding that design details matter and fragmentation will likely persist for now.
Uses historical case studies of aerospace, biotech and cryptography to model how states, firms and researchers will contest AI governance, predicting growing state control via tools like export controls.
A GovAI report offering a legal toolkit for drafting agreements that stay stable and keep their intent despite the radical, unpredictable changes advanced AI could bring.
Argues that AI risk analysis and policy should add a structural perspective, covering how competitive environments shape AI and its risks, beyond the misuse-accident dichotomy.
Argues that framing US-China AI development as an arms race encourages zero-sum thinking, and examines how that framing distorts policy choices on AI competition and risk.
Bostrom, Dafoe and Flynn derive a normative 'vector field' of policy desiderata for governing superintelligent AI, weighting considerations that matter more in long-term AI policy.
Allan Dafoe's GovAI research agenda sets out a framework for AI governance research, framing the overall problem and listing pivotal open questions with references.
A GovAI syllabus for self-study on the intersection of AI and international security, covering military AI, arms races and related governance risks.
A 2018 multi-institution report that forecasts security risks from malicious AI use across digital, physical, and political domains and proposes a research agenda for mitigation.
Bostrom's preliminary macrostrategy analysis weighs the global desirability of openness in AI development, including openness about safety techniques, capabilities and goals, finding short-term impacts mostly beneficial.