Open Problems in Chatbots and Mental Health
Reviews open problems in evaluating general-purpose chatbots for mental-health risks like sycophancy and delusion reinforcement, and argues for real-world data access and evidence-based policy.
Research
arXiv
Research preprints from the scientific community. arxiv.org
Focus All
Topic Lab practice + Evals
Source arXiv · 56
Date Any time
1 item
1 item
BMMS: Policy-Conditioned Measurement of Behavioral Compliance in Language Models
Introduces BMMS, a policy-conditioned suite that scores LLM compliance with published rulebooks, with an attribution-based admission test; finds GPT-4o fails 70.7% of its Model Spec hard constraints versus 0% and 9.1% for Claude models.
1 item
The Promises and Limitations of Evaluation-Based AI Governance: Evidence from Expert Interviews
Interviews with 37 experts reconstruct how frontier AI evaluations are expected to reduce societal risk, finding key pathways and challenges, including diminishing informativeness and companies' control over outcomes.
1 item
The Watchtower Imperative: Lifespan-Calibrated Evaluation for Frontier AI
Analyzes documented AI harms across the human lifespan and proposes lifespan design principles and a three-tier external evaluation framework for frontier AI safety.
1 item
The Moral Check: Strategic AI Governance for the Pacing Problem
Proposes an ex-ante governance framework (SAGE-X) for frontier AI with deterministic tripwires and a Moral Check Index, grounded in a PRISMA review of 130 empirical studies on evaluator confidence and guardrail decay.
1 item
Challenges of Auditing: Variability in Outputs of Large Language Models for Health
Shows that LLM health answers differ across access modes such as chatbot interfaces and APIs, so API-based evaluations may not validly audit consumer experiences.
1 item
Artificial intelligence and biosecurity: capabilities, threat pathways, and defense-in-depth governance
A review of AI-enabled biosecurity threats, evidence of uplift, limits of alignment transfer to biological models, interpretability for auditing removed capabilities, and defense-in-depth governance.
1 item
Toward a Threat Actor Profiling Taxonomy for Pre-Release Risk Management of Open-Weight Frontier Models
Proposes a six-attribute threat actor taxonomy so pre-release misuse evaluations of open-weight frontier models state adversary assumptions explicitly, comparably, and before testing.
1 item
A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communities
Introduces RepoComplianceBench to test whether coding agents follow open-source AI contribution rules; they rarely retrieve rules and never refuse in AI-banned repositories, even with prompting.}
1 item
Harmonizing AI Safety Thresholds
Proposes harmonized capability thresholds for frontier AI misuse risks (cyber, biological) and automated AI R&D, so third parties can verify crossings and safety standards don't race to the bottom.
1 item
Open Technical Problems in Open-Weight AI Model Risk Management
Lays out 16 open technical challenges for managing safety risks of frontier open-weight AI models across training data, training, evaluation, deployment and ecosystem monitoring.
1 item
The Governance of Human-LLM Interaction: Safety Gating, Civility Steering, and Affective Default Lock-In
Frames LLM communication style as a governance object and introduces a multi-agent pipeline measuring prompt steerability and style drift, showing provider alignment stabilizes affective defaults users cannot easily override.
1 item
Why Do AI Agents Break Rules? How Framing, Context, and Social Signals Shape Compliance
Tests twelve language-model procurement agents under regulatory pressure and finds large compliance failures from incentives and social pressure, which standard alignment benchmarks do not measure.
2 items
Muse Spark Safety & Preparedness Report
Meta's safety and preparedness report for Muse Spark evaluates catastrophic risks in chem-bio, cyber and loss of control, finds elevated pre-mitigation risk, and justifies release after safeguards.
Position: Behavioural Assurance Cannot Verify the Safety Claims Governance Now Demands
Position paper arguing behavioural evaluations cannot verify the safety claims AI governance frameworks require, formalising an audit gap and proposing mechanistic-evidence access as a fix.
1 item
TourMart: A Parametric Audit Instrument for Commission Steering in LLM Travel Agents
Proposes TourMart, a paired counterfactual audit instrument that measures how much commission-aware LLM travel agents steer recommendations toward higher-margin options, finding +3.5 to +7.7pp steering in tested models.
1 item
NeurIPS Should Require Reproducibility Standards for Frontier AI Safety Claims
Position paper arguing NeurIPS should require reproducibility standards and a tiered disclosure framework for frontier AI safety claims, since the artefacts needed to evaluate them are often withheld.
1 item
Code World Model Preparedness Report
Meta's preparedness report assesses the Code World Model for catastrophic-risk domains and misaligned propensities before open-weight release, concluding it adds no unusual frontier risk.
1 item
Safety Drift After Fine-Tuning: Evidence from High-Stakes Domains
Analyzing 100 models, the paper finds that benign fine-tuning causes inconsistent safety changes across benchmarks, so base-model safety assessments do not reliably transfer to deployed fine-tunes.
1 item
Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report v1.5
A technical report assessing frontier AI risks across cyber offense, persuasion, strategic deception, uncontrolled R&D, and self-replication, and proposing mitigation strategies for secure deployment.
1 item
Evaluating Nova 2.0 Lite model under Amazon's Frontier Model Safety Framework
Reports an evaluation of Amazon's Nova 2.0 Lite under its Frontier Model Safety Framework across CBRN, offensive cyber and automated AI R&D risk domains using benchmarks, red-teaming and uplift studies.
1 item
Expanding External Access To Frontier AI Models For Dangerous Capability Evaluations
Proposes a taxonomy of evaluator access levels (model access, information, timeframe) for dangerous capability evaluations of frontier AI, linking them to EU Code of Practice requirements.
1 item
Just Asking Questions: Doing Our Own Research on Conspiratorial Ideation by Generative AI Chatbots
Audits six commercial AI chatbots' safety guardrails against conspiracy theories, finding guardrail strength varies widely by model and topic, with selective focus on racism and national trauma.
1 item
Large language models require a new form of oversight: capability-based monitoring
Perspective arguing that LLMs deployed in healthcare need capability-based monitoring, organising oversight around shared model capabilities to catch systemic weaknesses, long-tail errors and emergent behaviours.
1 item
What do model reports say about their ChemBio benchmark evaluations? Comparing recent releases to the STREAM framework
Compares the ChemBio safety evaluation reporting in o3, Claude 4 and Gemini 2.5 Pro model reports against the STREAM standard, finding gaps in transparency across all three.
1 item
STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports
Proposes STREAM, a reporting standard for disclosing chemical and biological dangerous-capability evaluation results in AI model reports, developed with 23 experts and including a template.
1 item
Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report
Assesses frontier AI models across seven risk areas, including cyber, bio, scheming and self-replication, using red and yellow lines; finds no model crosses a red line.
2 items
Red Teaming AI Red Teaming
Critiques current AI red teaming as too focused on model-level flaws and proposes a macro and micro framework covering sociotechnical systems, with six recommendations for multifunctional teams.
Evaluating the Critical Risks of Amazon's Nova Premier under the Frontier Model Safety Framework
Amazon evaluates its Nova Premier model against frontier safety thresholds in CBRN, offensive cyber, and automated AI R&D domains, using benchmarks, red-teaming, and uplift studies, concluding it is safe to release.
1 item
Attestable Audits: Verifiable AI Safety Benchmarks Using Trusted Execution Environments
Proposes Attestable Audits, running safety benchmarks inside trusted execution environments so users can verify compliant model evaluations while protecting model IP and audit data, with a Llama-3.1 prototype.
1 item
Safety by Measurement: A Systematic Literature Review of AI Safety Evaluation Methods
A systematic literature review of AI safety evaluation methods covering capabilities, propensities, control, governance integration, and challenges like sandbagging and safetywashing.
1 item
Audit Cards: Contextualizing AI Evaluations
Proposes audit cards, a structured format for reporting AI evaluation context such as auditor identity, scope, access and conflicts of interest, after finding most existing evaluation reports omit it.
1 item
AI Companies Should Report Pre- and Post-Mitigation Safety Evaluations
Position paper arguing frontier AI companies should disclose both pre- and post-mitigation safety evaluations to inform policymakers, citing gaps in current lab disclosures.
1 item
Intolerable Risk Threshold Recommendations for Artificial Intelligence
Proposes principles and specific intolerable-risk thresholds for frontier AI across eight risk categories, including CBRN, cyber, model autonomy and deception, to guide policymakers and industry commitments.
1 item
Assessing confidence in frontier AI safety cases
Applies the Assurance 2.0 methodology to a frontier AI cyber-misuse inability safety case, quantifying confidence via an LLM-run Delphi method and proposing ways to prioritise defeaters and communicate confidence.
1 item
Enabling External Scrutiny of AI Systems with Privacy-Enhancing Technologies
Describes OpenMined's privacy-enhancing technology infrastructure for privacy-preserving external audits of AI systems, with case studies including frontier model evaluation with the UK AI Safety Institute, and argues for policy support.
1 item
Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTs
Automated black-box red-teaming and LLM-as-judge evaluation of 782 GPT Store custom chatbots finds 58.7% produce policy-violating responses, mostly inherited from base models.
1 item
Towards Frontier Safety Policies Plus
Argues that Frontier Safety Policies should evolve into more granular FSPs Plus, using precursory-capability metrics from standardization bodies and a feedback loop with AI safety cases.
1 item
Dynamic safety cases for frontier AI
Proposes a dynamic safety case management system for frontier AI that semi-automatically updates safety arguments post-deployment, demonstrated on an offensive cyber capabilities template with governance integration.
1 item
AI red-teaming is a sociotechnical problem: on values, labor, and harms
Argues that AI red-teaming is a sociotechnical practice and calls for computer and social scientists to study its values, labor arrangements, and psychological harms to red-teamers.