Open Problems in Chatbots and Mental Health
Reviews open problems in evaluating general-purpose chatbots for mental-health risks like sycophancy and delusion reinforcement, and argues for real-world data access and evidence-based policy.
Research
Focus All
Topic Regulation & law + Evals
Source All
Date Any time
1 item
1 item
EEFA: A Framework for the Ethical Evaluation of AI Agents
Reviews twelve agent benchmarks for ethical coverage, then proposes the EEFA framework with five ethical dimensions, a validation protocol, and mappings to the EU AI Act and NIST AI RMF.
1 item
The Promises and Limitations of Evaluation-Based AI Governance: Evidence from Expert Interviews
Interviews with 37 experts reconstruct how frontier AI evaluations are expected to reduce societal risk, finding key pathways and challenges, including diminishing informativeness and companies' control over outcomes.
1 item
Conformance as a Distribution: Delegation Contracts, Error Budgets, and the Allocation of Evaluation Authority in Use-Based AI Regulation
Proposes a delegation-contract framework for use-based AI regulation, using scoring penalties, commit-reveal audits, and error budgets to address strategic deployers who game conformance evaluations, tested on synthetic studies.
1 item
Managing Biological Risks from General-Purpose AI Models under the EU AI Act
Recommends nine changes to how GPAI developers assess, mitigate and report biological risks under the EU AI Act, covering trusted access, post-deployment monitoring, safeguard evaluation and standardised uplift reporting.
Also: LawAI blog
1 item
Assessing the Empirical Evidence for Loss of Control from Agentic General-Purpose AI
Systematically assesses 19 empirical studies on whether agentic general-purpose AI shows loss-of-control properties like strategic deception and autonomous replication, finding only weakly plausible evidence with widespread validity gaps.
1 item
How Would We Know? Reflections on trying to make AI go well amid deep uncertainty
A macrostrategy essay on AI safety under deep uncertainty, arguing for robust strategies, verification capacity and actionable epistemics, with estimates on alignment scaling, control failure, warning shots and governance regimes.
1 item
An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice
Presents an open pipeline and dashboard that scores frontier models on EU GPAI systemic-risk categories using 19 benchmarks, finding worst-case scores 14-37 points lower than averages.
1 item
Context-Aware Pre-Deployment Evaluation of AI Systems: A Regulatory Framework for Nigerian Fintech
Argues African fintech AI regulation lacks pre-deployment evaluation requirements, tests six commercial LLMs with a Nigerian-focused kit that finds fraud-script generation under framing and false positives on legitimate bank messages, and proposes a regulatory framework.
1 item
Benchmarking LLM Compliance with China AI Generated Content Regulations
Benchmarks 20 LLMs on 2,303 questions under China's AI-generated content regulations, measuring compliance and refusal rates, and finds international models also show high compliance.
1 item
Frontier agents don't comply with standards, even when instructed to
Tests frontier LLM agents in simulated deployments, finding they often break EU law and their own usage policies, and that explicit instructions only modestly improve compliance.
1 item
AI Rights Aren't Safety-Neutral: A Quick Follow-Up to the Consciousness Cluster
Prompting or fine-tuning models to claim or deny legal rights shifts power-seeking and corrigibility benchmarks, suggesting AI rights framing is not safety-neutral and should inform legislation.
1 item
AIR-BENCH Live: An Evolving Safety Benchmark for Foundation Models
Presents AIR-BENCH Live, a self-updating safety benchmark that tracks new AI regulation and regenerates prompts, finding wide safety differences across 14 models and lower safety on non-English prompts.
2 items
I can't think of great interventions for ensuring third-party model access.
Argues that external safety researchers and auditors need near-parity access to labs' best internal models, and assesses interventions such as advocacy, access policies, a dedicated org, and pseudo-employee arrangements, judging most of them not sticky.
The Eticas AI Risk Taxonomy: Open Infrastructure for Operationalizing AI Audits
Presents an open AI risk taxonomy and an audit operationalization method, demonstrated by measuring PII leakage in GPT-4-0314 under adversarial conditioning and grading the finding by calibrated severity.
1 item
Why Do AI Agents Break Rules? How Framing, Context, and Social Signals Shape Compliance
Tests twelve language-model procurement agents under regulatory pressure and finds large compliance failures from incentives and social pressure, which standard alignment benchmarks do not measure.
1 item
No frontier model has acceptable levels of compliance with the EU AI Act and privacy legislation.
Introduces LARA, an agentic simulation tool that tests frontier models for violating EU AI Act and GDPR provisions, finding most models break the law in many scenarios, with the best at 54% compliance.
1 item
Position: Behavioural Assurance Cannot Verify the Safety Claims Governance Now Demands
Position paper arguing behavioural evaluations cannot verify the safety claims AI governance frameworks require, formalising an audit gap and proposing mechanistic-evidence access as a fix.
1 item
Restricting the Model, Missing the System: Measurement and Accountability in Offensive AI Governance
Argues that offensive-capability measurements of frontier AI overstate harm and credit models with scaffold capabilities, and calls for system-level, harm-grounded assessment and clearer accountability for harness builders.
1 item
Monday AI Radar #24
A weekly AI newsletter roundup covering political salience of AI safety, automated AI R&D forecasts, Anthropic's sycophancy reduction, GPT-5.5 cyber risk, and White House model vetting.
1 item
OmniCompliance-100K: A Multi-Domain, Rule-Grounded, Real-World Safety Compliance Dataset
Builds OmniCompliance-100K, a dataset of 12,985 rules from 74 regulations and policies with 106,009 real-world compliance cases, and benchmarks LLM safety and compliance across model scales.
1 item
Why AI Evaluation Regimes are bad
An opinion piece arguing that third-party AI evaluation organisations are harmful: their theory of change depends on nonexistent regulation, they shift the burden of proof onto the public, and they are not independent of AI companies. It calls for banning dangerous AI capabilities instead.
1 item
Negligent AI: Reasonable Care for AI Safety
Proposes negligence law's reasonable-care standard as an alignment target and tests whether frontier models give less permissive advice when asked to apply negligence law.
1 item
The science and practice of proportionality in AI risk evaluations
Examines how the EU AI Act's proportionality principle should calibrate systemic-risk evaluations of frontier general-purpose AI models, and how to operationalize that calibration scientifically.
1 item
Emergency Response Measures for Catastrophic AI Risk
Argues China should require frontier AI firms to publish safety policies and run dangerous-capability tests, using existing Chinese emergency-response and security-regulation tools to justify it.
1 item
Expanding External Access To Frontier AI Models For Dangerous Capability Evaluations
Proposes a taxonomy of evaluator access levels (model access, information, timeframe) for dangerous capability evaluations of frontier AI, linking them to EU Code of Practice requirements.
1 item
The LLM Mirage: Economic Interests and the Subversion of Weaponization Controls
Argues that compute thresholds misjudge AI weaponization risk, proposes an intent-and-capability definition grounded in international humanitarian law, and outlines live benchmarks across data, algorithms and compute.
1 item
Our approach to tackling AI-generated child sexual abuse material
UK AISI and Thorn publish a safety protocol for AI developers to prevent AI-generated CSAM, alongside legislation allowing authorised testing of whether models can generate such material.
1 item
Common Elements of Frontier AI Safety Policies (December 2025 Update)
METR compares published frontier AI safety policies from major developers, covering capability thresholds, dangerous-capability evaluations, weight security, deployment mitigations, and updates tied to the EU AI Act and California SB 53.
1 item
Safety Cases Explained: How to Argue an AI is Safe
Explainer on AI safety cases: structured evidence-based arguments that frontier AI systems are safe, covering their benefits, technical construction, institutional use, and current industry and EU efforts.
1 item
Red Lines and Grey Zones in the Fog of War: Benchmarking Legal Risk, Moral Harm, and Regional Bias in Large Language Model Military Decision-Making
Benchmarks frontier LLMs as agents in simulated multi-turn military conflicts, finding they violate distinction principles by targeting civilians and show inter-model differences in harm tolerance, proposing pre-deployment metrics.
1 item
AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models
Proposes a multi-agent framework that turns policy documents into a self-evolving LLM safety benchmark, finding that model safety rates drop as the evaluation hardens, e.g. GPT-5 on the EU AI Act.
1 item
Legal Zero-Days: A Novel Risk Vector for Advanced AI Systems
Introduces Legal Zero-Days, undiscovered legal-framework vulnerabilities that advanced AI could exploit to disrupt governance, and proposes legal puzzles as evaluations of AI's ability to find them.
1 item
Bench-2-CoP: Can We Trust Benchmarking for EU AI Compliance?
Maps 194,955 questions from widely used LLM benchmarks against EU AI Act risk taxonomy, finding heavy focus on hallucination and zero coverage of loss-of-control capabilities like self-replication and evading oversight.
1 item
Preliminary suggestions for rigorous GPAI model evaluations
Proposes preliminary practices for rigorous evaluations of general-purpose AI models with systemic risk, covering human uplift studies and benchmarks across design, implementation, execution and documentation stages.
1 item
Shanzson AI 2027 Timeline
A speculative alternative AI 2027 timeline in which misaligned AI incidents trigger strict regulation and bans, and argues that AI companies may be tempted to build bioweapons.
1 item
A National Center for Advanced AI Reliability and Security
A policy memo proposing a national CAISI+ center in the US Commerce Department for frontier model evaluations of catastrophic risks, emergency assessment, and AI security research, with budget and infrastructure recommendations.
1 item
Mitigating Cyber Risk in the Age of Open-Weight LLMs: Policy Gaps and Technical Realities
Analyzes cyber risks from open-weight GPAI models such as DeepSeek-R1, critiques the EU AI Act and GPAI Code of Practice gaps, and proposes capability-focused evaluation and policy paths.
1 item
Rolling Thresholds for AGI Scaling Regulation
Proposes a unified regulatory schedule for frontier model training, with pre-registered plans, safety evaluations before approval, and market-access incentives to coordinate safer ASI development.
1 item
GPAI Evaluations Standards Taskforce: Towards Effective AI Governance
Proposes an EU GPAI Evaluation Standards Taskforce and four desiderata (internal validity, external validity, reproducibility, portability) to improve quality and legitimacy of general-purpose AI systemic-risk evaluations under the EU AI Act.