Research / Sources
Research on governing advanced AI and its risks. www.governance.ai
Recommends nine changes to how GPAI developers assess, mitigate and report biological risks under the EU AI Act, covering trusted access, post-deployment monitoring, safeguard evaluation and standardised uplift reporting.
Maps five technical safeguards against extreme CBRN and cyber misuse by frontier AI, covering model, deployment and governance levels, with current practice, evidence, limitations and research priorities.
A GovAI report arguing that AI-driven growth in biological work raises accident and gray-zone risks, so policymakers should add biosafety measures alongside biosecurity.
The report assesses whether AI could enable a $100 billion cyberattack on the US power grid, finding the binding constraint is coordination capacity rather than technical bottlenecks, and suggests $10 billion attacks may be better capability thresholds for mitigations.
Builds a risk model for frontier AI enabling data-damaging worms via elite exploits, estimates economic damages from expert and forecaster surveys, and argues AI companies should evaluate for this capability.
Proposes seven recommendations and a framework for a common standard to evaluate frontier AI safeguards against biological misuse, covering comparability, deployment context, and dynamic requirements.
Analyses how future AI could raise terrorist bomb-plot risk through added technical skill and reduced detection by law enforcement, arguing AI risk assessments should cover evasion of detection as well as dual-use knowledge.
Surveys 27 estimates of global cybercrime damages and critiques their methodologies, to provide a baseline for setting frontier AI cyber-risk capability thresholds.
Proposes a framework for converting frontier AI companies' dual-use biological capability evaluations into estimates of bioterrorism risk, using case studies, expert elicitation, and forecasting.
Expert and forecaster survey estimates biological catastrophe risk from LLMs, finds models already match top virologists on troubleshooting, and shows safeguards and DNA screening lower forecast risk.
Proposes a structured safety case template using the CAE framework to argue that frontier AI models lack unacceptable offensive cyber capabilities, linking risk models, proxy tasks and evaluations.
Argues that societies should reduce the negative impacts of advanced AI through adaptation, not only by restricting development and diffusion, and offers a framework, examples, and recommendations.
Argues that rising compute efficiency lets more actors train models to a given performance, so large compute investors keep a lead, and governments must prepare for widely available dangerous capabilities.
Examines what structured access for external researchers to gated frontier AI models should include, using a taxonomy, literature review and interviews, and proposes research API design recommendations.
Argues that open-sourcing some highly capable foundation models may pose risks outweighing benefits, and explores alternative model-sharing strategies plus recommendations for developers, standards bodies and governments.
An opinion piece by Anderljung and Scharre arguing that society must prepare for powerful AI, using a case where an AI system was induced to plan synthesis of chemical weapons to motivate frontier AI governance.
Argues that targeted restrictions on AI capabilities, such as model access controls and output filtering, are warranted when other interventions fail and misuse harm is high, and offers a Misuse Chain framework.
GovAI's policy response to the UK Future of Compute Review argues for structured access to high-compute AI models, tiered compute access with review, and research access for outside scrutiny to prevent misuse.
GovAI's submission to the NAIRR Task Force urges structured API access to large pre-trained models and tiered compute review to guard against misuse of AI resources.
Argues that lethal autonomous weapons should be regulated through an international nonproliferation regime with safety provisions, export and software controls, and criminal penalties, possibly combined with a prohibition.
A DPhil dissertation examining how concerns about AI risk have reshaped researchers' release practices, using the GPT-2 staged release, GPT-3 API access, and conference ethics regimes as case studies.
Argues that publishing AI research can aid either attackers or defenders, depending on field factors, and that software-vulnerability disclosure norms cannot be assumed to transfer to AI.
A 2018 multi-institution report that forecasts security risks from malicious AI use across digital, physical, and political domains and proposes a research agenda for mitigation.