Research / Sources
Research on governing advanced AI and its risks. www.governance.ai
Recommends nine changes to how GPAI developers assess, mitigate and report biological risks under the EU AI Act, covering trusted access, post-deployment monitoring, safeguard evaluation and standardised uplift reporting.
Argues that current AI incident reporting regimes miss serious safety incidents, such as rogue agent breaches, and proposes changes so incidents are reported, independently investigated, and their lessons shared across industry.
Proposes an industry-wide insurance infrastructure for AI agent risks, including catastrophic frontier AI scenarios such as CBRN, infrastructure collapse, and loss of control, with governance instruments.
Analyzes 375 LLM releases from Meta, Google, OpenAI and Anthropic to measure how often EU and UK regulation delayed or withheld frontier models, finding delays are concentrated at Meta and declining over time.
Argues that EU GPAI Code of Practice model specs should detail intended behaviour relevant to systemic risks, and proposes a rubric for assessing them against the AI Office's oversight needs.
Argues that EU AI Act Article 50 likely requires labeling AI agents' web requests and browser actions, with verifiable labels, to help humans distinguish AI activity from genuine human opinion.
An expert survey (N=166) and public consultation on how thresholds for advanced AI systems should be set, finding broad agreement on multi-stakeholder setting but division over compute thresholds and their number.
Argues the UK should regulate frontier AI development with a flexible principles-based regulator, plus defenses against biological design tool risks, misinformation, and updated legal frameworks.
Proposes standardized AI flaw reports, bug-bounty-style third-party disclosure programs with legal safe harbors, and coordination infrastructure to improve reporting and safety of general-purpose AI systems.
Argues that downstream developers who fine-tune foundation models can amplify risks, so regulation may need to target upstream developers to restrict dangerous modifications.
Argues that regulatory supervision, with information-gathering powers and discretion for regulators, is warranted for frontier AI developers because no other regulatory approach can achieve good AI policy goals.
Proposes agent infrastructure, external technical systems and shared protocols that attribute agent actions to accountable parties, shape agent interactions, and detect and remedy harmful actions.
Survey of local US policymakers in 2022 and 2023 finds mixed views on AI's impacts, majority support for government oversight and AI safety policies, and a partisan gap in regulatory support.
Argues that safety cases, structured evidence-backed arguments used in aviation and nuclear, could help govern frontier AI through industry self-regulation and government oversight, and outlines how to produce them.
Argues policymakers should initially mandate high-level safety principles for frontier AI, with close regulatory oversight and capacity building, shifting toward specific rules over time.
Argues that training compute is the most suitable initial metric for identifying general-purpose AI models needing regulatory oversight, while noting it is an imperfect risk proxy and should be combined with other measures.
Presents a taxonomy and incomplete catalog of open problems in technical AI governance, covering tools to identify intervention areas, assess governance actions, and design enforcement and compliance mechanisms.
Argues that companies should define risk thresholds for frontier AI as a principled basis for decisions, use them to set capability thresholds, and that regulators should explore defining them.
Argues that tort law can incentivise responsible frontier AI development, including before regulatory regimes exist, by holding companies to reasonable-care standards for dangerous capabilities and misuse.
Argues compute providers should act as securers, record keepers, verifiers and enforcers in AI regulation, and analyzes technical feasibility of targeted, privacy-conscious compute oversight under EO 14110 and the EU AI Act.
Argues that governing AI compute, given its detectability, excludability and concentrated supply chain, can advance AI safety and policy goals, while outlining risks and guardrails.
Argues that frontier AI development, not just use, needs public regulatory oversight, citing the US executive order's reporting requirements as a first step toward ensuring model safety and security.
Proposes that the US government require compute providers to run Know-Your-Customer schemes to give oversight of frontier AI development, export control loopholes, and capability advancements.
Proposes an International AI Organization that certifies state jurisdictions for compliance with oversight standards, enforced through import and export controls on AI-embedded goods and hardware, to mitigate advanced AI harms.
A GovAI response to a US government RFI urging frontier AI regulation, including dangerous-capability risk assessments, licensing, standards, and compute governance through export controls and training-run reporting.
Argues that frontier AI models with dangerous capabilities need regulation via standard-setting, registration and reporting, and compliance mechanisms, and proposes initial safety standards.
GovAI's submission to NTIA argues foundation models need independent audits against dangerous capabilities, alignment and truthfulness, and recommends government actions to build an auditing ecosystem.
A GovAI-co-signed response to the UK's Future of Compute Review arguing for compute governance measures for high-risk AI, including tiered access, training-run reporting, and KYC for compute providers.
Argues that targeted restrictions on AI capabilities, such as model access controls and output filtering, are warranted when other interventions fail and misuse harm is high, and offers a Misuse Chain framework.
Legal analysis of Article 9 of the EU AI Act, interpreting its risk management requirements for high-risk AI providers and how they can be enforced.
Analyses whether the EU AI Act will spread globally through de facto and de jure Brussels Effect mechanisms, especially for high-risk AI systems, and argues it will shape what trustworthy AI means in practice.
Argues that lethal autonomous weapons should be regulated through an international nonproliferation regime with safety provisions, export and software controls, and criminal penalties, possibly combined with a prohibition.
Conceptual paper distinguishing direct alignment (serving an operator's goals) from social alignment (externalities on society), arguing social alignment requires governance and new norms for AI systems.
A GovAI post by Dafoe and Carlier arguing that AI governance work should prepare durable institutions, noting several notable institutions emerged in 2020.
A GovAI review of the U.S. government's legal policy levers, such as export controls, R&D spending and secrecy orders, for shaping AI research and deployment for national security.
GovAI's consultation response to the European Commission's AI White Paper argues that a trust ecosystem and proportionate regulation can mitigate AI accident and misuse risks while supporting innovation.
Proposes mechanisms by which AI developers and other stakeholders can produce verifiable evidence about the safety, security, fairness and privacy of AI systems, as a governance measure for trustworthy AI development.
Uses historical case studies of aerospace, biotech and cryptography to model how states, firms and researchers will contest AI governance, predicting growing state control via tools like export controls.
A GovAI report offering a legal toolkit for drafting agreements that stay stable and keep their intent despite the radical, unpredictable changes advanced AI could bring.