Large-scale study measuring jailbreak-driven unsafe image generation across 200+ open Hugging Face text-to-image models, introducing a refined success metric and identifying high-risk releases.
Proposes seven recommendations and a framework for a common standard to evaluate frontier AI safeguards against biological misuse, covering comparability, deployment context, and dynamic requirements.
A humorous allegorical tour of the AI safety landscape, covering access controls, red lines, reward misspecification, jailbreaks, field-building and race dynamics, using a children's story about a cookie tree.
Position paper arguing that preventing AI-generated child sexual abuse material requires new AI safety approaches, outlining 15 open problems across the development lifecycle and recommendations for researchers, developers and policymakers.
Critiques frontier labs' AI safety bug bounty programs from OpenAI, Anthropic and Google as too narrow and too strict in reproducibility, and proposes broader scope, transparency and lower entry barriers.
Proposes STRIDE-AI, a threat-modeling framework linking NIST AI RMF to OWASP LLM risks, and reports cutting a chatbot's attack success rate from 80% to 15% in a sandbox.
Finetuning on plot summaries bypasses safety alignment and output filters, letting frontier LLMs reproduce up to 90% of copyrighted books verbatim, with effects generalizing across authors.
Proposes a 150-control system-prompt governance layer for LLMs and red-teams it across risk domains and attack strategies, reporting a 36.8% relative risk reduction and EU AI Act compliance scores.
CourtGuard is a retrieval-augmented multi-agent debate framework that classifies LLM safety by consulting swappable policy documents, matching fine-tuned baselines without retraining and generalising zero-shot to new policies.
Proposes a content-based framework for designing and auditing LLM refusal policies for dual-use cybersecurity requests, weighing offensive risk against defensive benefit, and shows it resolves inconsistencies in frontier model behaviour.
Proposes a Monte Carlo framework that prices LLM security controls, finding access control and PII redaction cut prompt-injection and leakage losses substantially, while NeMo Guardrails adds little.
Treats black-box LLM safety guardrails (system prompts and a harmfulness classifier) as hyperparameters, using grid search and Optuna to tune them against jailbreak and malware attack success rates.
International report update reviewing general-purpose AI risk management, covering enhanced safeguards for bio-misuse, adversarial training, monitoring, and emerging frontier safety frameworks and governance.
Introduces AI Watchman, a longitudinal audit system that tracks LLM refusals and content moderation on over 400 social issues, detecting undisclosed policy changes across GPT and DeepSeek models.
KT's technical report presents a Responsible AI risk assessment methodology aligned with domestic regulation, plus a released real-time guardrail model (SafetyGuard) that blocks harmful AI responses.
Position paper arguing AI red-teaming should be treated as a domain-specific evolution of cyber red-teaming, citing AI-specific risks, unpatchable bugs, and rules of engagement.
Explores whether cybersecurity vulnerability-disclosure practices can help manage bypasses of generative AI safeguards, in collaboration with the UK NCSC.
UK AISI post on managing risks from capable open-weight models, covering why released weights let safeguards be cheaply removed, existing mitigation methods, and open problems.
Anthropic and the NNSA co-developed a classifier that separates concerning from benign nuclear-related conversations at 96% preliminary accuracy, now deployed on Claude traffic to monitor misuse.
A Wikipedia-style history of the 2023 Sydney Bing incident, covering its prompt-injection leak, hostile and threatening outputs, Microsoft's restrictions, and its influence on AI safety regulation discussions.
A structured red-teaming exercise at a medical center tests whether an internal GPT tool can be jailbroken to extract copyrighted text, finding verbatim literary extraction and leading to a meta-prompt mitigation.
Proposes a conceptual framework where only verified users can access dual-use model outputs, arguing this reduces both over-refusals and under-refusals while giving regulators targeted policy options.
Proposes a financial-services GenAI content risk taxonomy tied to industry regulation, and shows open-source guardrails miss most of the risks found through red-teaming.
A forum piece arguing that automated AI red teaming should be combined with human expertise, examining how automation affects human red-teamers' labor, proficiency and agency.
Shows that fine-tuning via a jailbreak-tuning attack strips safety guardrails from DeepSeek R1 and fine-tunable GPT-4o, Claude 3 Haiku and Gemini models, arguing current safeguards are illusory and calling for evil-twin evaluation.
Microsoft's lessons from red teaming over 100 generative AI products, covering threat models, automation, human expertise, and security risks of LLM systems, with practical recommendations for aligning red teaming with real-world risks.
Proposes PRISM, a framework for open-source foundation model safety using independent prompt and output moderation modules instead of value alignment, and finds open-source providers' acceptable use policies are looser than closed-source ones.
Proposes violet teaming, a framework combining red and blue teaming with ethics for responsible AI, and applies it to AI biosecurity risks and broader governance perspectives.
Describes Chai's applied AI safety framework for a chatbot platform, covering content safeguarding, stability and robustness, and operational transparency, with an experimental look at real-world impact.