Research / Sources
Research on governing advanced AI and its risks. www.governance.ai
Recommends nine changes to how GPAI developers assess, mitigate and report biological risks under the EU AI Act, covering trusted access, post-deployment monitoring, safeguard evaluation and standardised uplift reporting.
Argues that frontier AI developers should host embedded third-party assessments giving evaluators internal access to agents, security controls and alignment, with continuous assessment and quarterly reports.
Maps five technical safeguards against extreme CBRN and cyber misuse by frontier AI, covering model, deployment and governance levels, with current practice, evidence, limitations and research priorities.
A GovAI report arguing that AI-driven growth in biological work raises accident and gray-zone risks, so policymakers should add biosafety measures alongside biosecurity.
Proposes an industry-wide insurance infrastructure for AI agent risks, including catastrophic frontier AI scenarios such as CBRN, infrastructure collapse, and loss of control, with governance instruments.
Analyzes 375 LLM releases from Meta, Google, OpenAI and Anthropic to measure how often EU and UK regulation delayed or withheld frontier models, finding delays are concentrated at Meta and declining over time.
Proposes seven recommendations and a framework for a common standard to evaluate frontier AI safeguards against biological misuse, covering comparability, deployment context, and dynamic requirements.
Position paper arguing frontier AI governance must go beyond model-level controls, because inference, scaffolding and asset-based capability gains can undermine pre-deployment evaluation and mitigation.
Argues that EU GPAI Code of Practice model specs should detail intended behaviour relevant to systemic risks, and proposes a rubric for assessing them against the AI Office's oversight needs.
Proposes frontier AI auditing: third-party verification of developers' safety and security claims using non-public access, with four AI Assurance Levels and recommendations for baseline and near-term adoption.
Argues that frontier AI companies lowering safety mitigations based on competitors' models can increase total risk, since competitor capabilities are hard to assess and erosion of standards may follow.
Proposes STREAM, a reporting standard with templates and examples to help frontier AI developers transparently disclose chemical and biological dangerous-capability evaluation results in model reports.
Proposes a systems-safety framework for analysing AI agent incidents, identifying system, contextual and cognitive causal factors, and recommending what incident reports and developer logs should contain.
Argues the UK should regulate frontier AI development with a flexible principles-based regulator, plus defenses against biological design tool risks, misinformation, and updated legal frameworks.
Proposes third-party compliance reviews to verify that frontier AI companies follow their safety frameworks, weighing benefits, risks, and practical design options for reviewers, disclosure, and timing.
Proposes standardized AI flaw reports, bug-bounty-style third-party disclosure programs with legal safe harbors, and coordination infrastructure to improve reporting and safety of general-purpose AI systems.
Argues that downstream developers who fine-tune foundation models can amplify risks, so regulation may need to target upstream developers to restrict dangerous modifications.
Argues that regulatory supervision, with information-gathering powers and discretion for regulators, is warranted for frontier AI developers because no other regulatory approach can achieve good AI policy goals.
Argues that safety cases, structured evidence-backed arguments used in aviation and nuclear, could help govern frontier AI through industry self-regulation and government oversight, and outlines how to produce them.
Proposes assigning identifiers to individual AI system instances so users, investigators and operators can verify certifications, trace incidents and shut down malfunctioning systems, and analyses adoption incentives and risks.
Proposes a grading rubric with seven criteria and 21 indicators for evaluating frontier AI companies' safety frameworks, along with methods for applying it such as surveys, Delphi studies, and audits.
Argues policymakers should initially mandate high-level safety principles for frontier AI, with close regulatory oversight and capacity building, shifting toward specific rules over time.
Argues that companies should define risk thresholds for frontier AI as a principled basis for decisions, use them to set capability thresholds, and that regulators should explore defining them.
Argues that tort law can incentivise responsible frontier AI development, including before regulatory regimes exist, by holding companies to reasonable-care standards for dangerous capabilities and misuse.
GovAI's response to NIST's RFI recommends proportionate risk management, safety policies, evaluations, red-teaming and post-deployment monitoring for dual-use foundation models.
Argues that frontier AI development, not just use, needs public regulatory oversight, citing the US executive order's reporting requirements as a first step toward ensuring model safety and security.
Surveys six requirements for effective external scrutiny of frontier LLMs (red-teaming, auditing, researcher access) under the ASPIRE framework and recommends policy measures.
Examines what structured access for external researchers to gated frontier AI models should include, using a taxonomy, literature review and interviews, and proposes research API design recommendations.
Proposes adapting the three lines of defense risk management model to AI companies, to clarify risk responsibilities, close coverage gaps, and strengthen board oversight of AI risks.
Proposes a five-step scheme in which frontier developers pause R&D when dangerous-capability evaluations fail, and compares voluntary, collective, auditor-based, and legally required versions.
Argues that open-sourcing some highly capable foundation models may pose risks outweighing benefits, and explores alternative model-sharing strategies plus recommendations for developers, standards bodies and governments.
Reviews risk identification, analysis and evaluation techniques from safety-critical industries and recommends how AGI companies could use them to assess catastrophic AI risks, while noting they are insufficient alone.
Argues that frontier AI models with dangerous capabilities need regulation via standard-setting, registration and reporting, and compliance mechanisms, and proposes initial safety standards.
Proposes a three-layer auditing framework for LLMs covering provider governance, pre-release model audits, and application audits, and discusses what auditing can realistically achieve.
Argues that dangerous-capability and alignment evaluations are essential for identifying extreme risks from general-purpose AI and informing policymakers and decisions on training, deployment and security.
A survey of 92 AGI experts (51 responses) on 50 statements about what AGI labs should do; participants on average agreed with all, with near-unanimous support for pre-deployment risk assessment, dangerous-capability evals, and third-party audits.
Examines how AI companies can design ethics boards to reduce AI risks, covering responsibilities, legal structure, membership, decision-making and resources, and offers a design toolbox.
Argues that privacy-enhancing technologies could support AI governance tasks like external scrutiny, auditing and source verification, and that these should be designed as an interoperable system.
Legal analysis of Article 9 of the EU AI Act, interpreting its risk management requirements for high-risk AI providers and how they can be enforced.
Presents an interactive web app for a formal safety-performance tradeoff model of AI competition, examining why safety breakthroughs may not yield safer deployed systems.