Embedded Assessments for Frontier AI
Argues that frontier AI developers should host embedded third-party assessments giving evaluators internal access to agents, security controls and alignment, with continuous assessment and quarterly reports.
Research / Sources
GovAI
Research on governing advanced AI and its risks. www.governance.ai
Focus All
Topic Evals & standards
Date Any time
Improving Frontier AI Incident Reporting Regimes
Argues that current AI incident reporting regimes miss serious safety incidents, such as rogue agent breaches, and proposes changes so incidents are reported, independently investigated, and their lessons shared across industry.
Technical Safeguards Against Extreme Misuse of AI: Current Landscape and Future Directions
Maps five technical safeguards against extreme CBRN and cyber misuse by frontier AI, covering model, deployment and governance levels, with current practice, evidence, limitations and research priorities.
Underwriting the Agent Economy: The Blueprint for an AI Insurance Stack
Proposes an industry-wide insurance infrastructure for AI agent risks, including catastrophic frontier AI scenarios such as CBRN, infrastructure collapse, and loss of control, with governance instruments.
Assessing the Risk of AI-Enabled Computer Worms
Builds a risk model for frontier AI enabling data-damaging worms via elite exploits, estimates economic damages from expert and forecaster surveys, and argues AI companies should evaluate for this capability.
Towards a Common Standard for Evaluating Frontier AI Safeguards Against Biological Misuse
Proposes seven recommendations and a framework for a common standard to evaluate frontier AI safeguards against biological misuse, covering comparability, deployment context, and dynamic requirements.
Comprehensive AI Governance Requires Addressing Non-Model Capability Gains
Position paper arguing frontier AI governance must go beyond model-level controls, because inference, scaffolding and asset-based capability gains can undermine pre-deployment evaluation and mitigation.
Measuring AI R&D Automation
Proposes metrics, such as capital share of AI R&D spending and AI subversion incidents, to track AI R&D automation and its effects on safety progress and oversight, and recommends companies and governments adopt them.
Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies
Proposes frontier AI auditing: third-party verification of developers' safety and security claims using non-public access, with four AI Assurance Levels and recommendations for baseline and near-term adoption.
Dual-Use AI Capabilities and the Risk of Bioterrorism
Proposes a framework for converting frontier AI companies' dual-use biological capability evaluations into estimates of bioterrorism risk, using case studies, expert elicitation, and forecasting.
Permission Manifests for Web Agents
Proposes agent-permissions.json, a robots.txt-style manifest letting websites specify allowed LLM web-agent interactions, to enable compliant agent access while respecting site owners' preferences.
STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports
Proposes STREAM, a reporting standard with templates and examples to help frontier AI developers transparently disclose chemical and biological dangerous-capability evaluation results in model reports.
Survey on Thresholds for Advanced AI Systems
An expert survey (N=166) and public consultation on how thresholds for advanced AI systems should be set, finding broad agreement on multi-stakeholder setting but division over compute thresholds and their number.
Incident Analysis for AI Agents
Proposes a systems-safety framework for analysing AI agent incidents, identifying system, contextual and cognitive causal factors, and recommending what incident reports and developer logs should contain.
Forecasting LLM-Enabled Biorisk and the Efficacy of Safeguards
Expert and forecaster survey estimates biological catastrophe risk from LLMs, finds models already match top virologists on troubleshooting, and shows safeguards and DNA screening lower forecast risk.
Third-Party Compliance Reviews for Frontier AI Safety Frameworks
Proposes third-party compliance reviews to verify that frontier AI companies follow their safety frameworks, weighing benefits, risks, and practical design options for reviewers, disclosure, and timing.
In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI
Proposes standardized AI flaw reports, bug-bounty-style third-party disclosure programs with legal safe harbors, and coordination infrastructure to improve reporting and safety of general-purpose AI systems.
Infrastructure for AI Agents
Proposes agent infrastructure, external technical systems and shared protocols that attribute agent actions to accountable parties, shape agent interactions, and detect and remedy harmful actions.
Safety Case Template for Frontier AI: A Cyber Inability Argument
Proposes a structured safety case template using the CAE framework to argue that frontier AI models lack unacceptable offensive cyber capabilities, linking risk models, proxy tasks and evaluations.
Safety Cases for Frontier AI
Argues that safety cases, structured evidence-backed arguments used in aviation and nuclear, could help govern frontier AI through industry self-regulation and government oversight, and outlines how to produce them.
IDs for AI Systems
Proposes assigning identifiers to individual AI system instances so users, investigators and operators can verify certifications, trace incidents and shut down malfunctioning systems, and analyses adoption incentives and risks.
A Grading Rubric for AI Safety Frameworks
Proposes a grading rubric with seven criteria and 21 indicators for evaluating frontier AI companies' safety frameworks, along with methods for applying it such as surveys, Delphi studies, and audits.
Open Problems in Technical AI Governance
Presents a taxonomy and incomplete catalog of open problems in technical AI governance, covering tools to identify intervention areas, assess governance actions, and design enforcement and compliance mechanisms.
Visibility into AI Agents
Assesses agent identifiers, real-time monitoring and activity logging as ways to increase visibility into AI agents, analysing their governance value, implementation trade-offs and privacy and power-concentration implications.
Governing Through the Cloud
Argues compute providers should act as securers, record keepers, verifiers and enforcers in AI regulation, and analyzes technical feasibility of targeted, privacy-conscious compute oversight under EO 14110 and the EU AI Act.
Response to the RFI Related to NIST's Assignments Under the Executive Order Concerning AI
GovAI's response to NIST's RFI recommends proportionate risk management, safety policies, evaluations, red-teaming and post-deployment monitoring for dual-use foundation models.
Towards Publicly Accountable Frontier LLMs
Surveys six requirements for effective external scrutiny of frontier LLMs (red-teaming, auditing, researcher access) under the ASPIRE framework and recommends policy measures.
Structured Access for Third-Party Research on Frontier AI Models
Examines what structured access for external researchers to gated frontier AI models should include, using a taxonomy, literature review and interviews, and proposes research API design recommendations.
Coordinated Pausing: An Evaluation-Based Coordination Scheme for Frontier AI Developers
Proposes a five-step scheme in which frontier developers pause R&D when dangerous-capability evaluations fail, and compares voluntary, collective, auditor-based, and legally required versions.
National Priorities for Artificial Intelligence (Response to the OSTP Request for Information)
A GovAI response to a US government RFI urging frontier AI regulation, including dangerous-capability risk assessments, licensing, standards, and compute governance through export controls and training-run reporting.
Frontier AI Regulation: Managing Emerging Risks to Public Safety
Argues that frontier AI models with dangerous capabilities need regulation via standard-setting, registration and reporting, and compliance mechanisms, and proposes initial safety standards.
Response to the NTIA AI Accountability Policy
GovAI's submission to NTIA argues foundation models need independent audits against dangerous capabilities, alignment and truthfulness, and recommends government actions to build an auditing ecosystem.
Auditing Large Language Models: A Three‐Layered Approach
Proposes a three-layer auditing framework for LLMs covering provider governance, pre-release model audits, and application audits, and discusses what auditing can realistically achieve.
Model Evaluation for Extreme Risks
Argues that dangerous-capability and alignment evaluations are essential for identifying extreme risks from general-purpose AI and informing policymakers and decisions on training, deployment and security.
Towards Best Practices in AGI Safety and Governance
A survey of 92 AGI experts (51 responses) on 50 statements about what AGI labs should do; participants on average agreed with all, with near-unanimous support for pre-deployment risk assessment, dangerous-capability evals, and third-party audits.
Exploring the Relevance of Data Privacy-Enhancing Technologies for AI Governance Use Cases
Argues that privacy-enhancing technologies could support AI governance tasks like external scrutiny, auditing and source verification, and that these should be designed as an interoperable system.
Submission to the NIST AI Risk Management Framework
GovAI's comments on NIST's AI Risk Management Framework urge more focus on catastrophic risks, a misuse category, an alignment principle, internal audits, and frequent framework updates.
Filling gaps in trustworthy development of AI
A GovAI policy piece proposing verifiable mechanisms, such as audits and accountability tools, that let AI developers demonstrate trustworthiness and close the gap between ethics principles and practice.
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Proposes mechanisms by which AI developers and other stakeholders can produce verifiable evidence about the safety, security, fairness and privacy of AI systems, as a governance measure for trustworthy AI development.
Standards for AI Governance: International Standards to Enable Global Coordination in AI Research & Development
Argues that international standards bodies are an important, underused policy tool for coordinating beneficial AI research and development, and recommends engagement for researchers and AI organisations.