Argues that LLM safeguard metrics like refusal and attack success rates don't establish deployment-level safety, and that system-level evidence is needed to judge whether a safeguard makes a deployment safer.
Large-scale study measuring jailbreak-driven unsafe image generation across 200+ open Hugging Face text-to-image models, introducing a refined success metric and identifying high-risk releases.
Proposes seven recommendations and a framework for a common standard to evaluate frontier AI safeguards against biological misuse, covering comparability, deployment context, and dynamic requirements.
A humorous allegorical tour of the AI safety landscape, covering access controls, red lines, reward misspecification, jailbreaks, field-building and race dynamics, using a children's story about a cookie tree.
Position paper arguing that preventing AI-generated child sexual abuse material requires new AI safety approaches, outlining 15 open problems across the development lifecycle and recommendations for researchers, developers and policymakers.
Critiques frontier labs' AI safety bug bounty programs from OpenAI, Anthropic and Google as too narrow and too strict in reproducibility, and proposes broader scope, transparency and lower entry barriers.
Proposes STRIDE-AI, a threat-modeling framework linking NIST AI RMF to OWASP LLM risks, and reports cutting a chatbot's attack success rate from 80% to 15% in a sandbox.
Benchmarks jailbreak attacks (Baseline, BitBypass, DeepInception) on three commercial LLMs as grid-operations assistants, measuring violations of NERC reliability standards with attack success rates from 0% to 63%.
Argues that open-weight models undermine the defenses proposed for near-future AI risks, since guardrails can be fine-tuned away, use cannot be monitored, and bad actors have access by default.
Proposes a content-based framework for designing and auditing LLM refusal policies for dual-use cybersecurity requests, weighing offensive risk against defensive benefit, and shows it resolves inconsistencies in frontier model behaviour.
Position paper arguing capability control should be a separate goal from alignment, organising data, weight-level and system-level restrictions into a defense-in-depth approach against misuse of foundation models.
Proposes a Monte Carlo framework that prices LLM security controls, finding access control and PII redaction cut prompt-injection and leakage losses substantially, while NeMo Guardrails adds little.
Treats black-box LLM safety guardrails (system prompts and a harmfulness classifier) as hyperparameters, using grid search and Optuna to tune them against jailbreak and malware attack success rates.
International report update reviewing general-purpose AI risk management, covering enhanced safeguards for bio-misuse, adversarial training, monitoring, and emerging frontier safety frameworks and governance.
This perspective maps biosecurity threats from generative AI, including jailbreaks, privacy risks and autonomous agents, and draws on 130 expert interviews to call for governance frameworks and layered safeguards.
Position paper arguing AI red-teaming should be treated as a domain-specific evolution of cyber red-teaming, citing AI-specific risks, unpatchable bugs, and rules of engagement.
Explores whether cybersecurity vulnerability-disclosure practices can help manage bypasses of generative AI safeguards, in collaboration with the UK NCSC.
UK AISI post on managing risks from capable open-weight models, covering why released weights let safeguards be cheaply removed, existing mitigation methods, and open problems.
Anthropic and the NNSA co-developed a classifier that separates concerning from benign nuclear-related conversations at 96% preliminary accuracy, now deployed on Claude traffic to monitor misuse.
Proposes a cognitive cybersecurity framework (CIA+TA) and quantitative risk methodology for adversarial manipulation of AI reasoning, validated on prior studies and recommending pre-deployment penetration testing.
Introduces PoisonSwarm, which launders malicious tasks by splitting them across crowdsourced LLMs with different safety policies, outperforming prior methods and showing that ecosystem-level regulation is hard.
Proposes an end-to-end safety case showing misuse safeguards reduce AI assistant misuse risk to low levels, combining red-teaming effort estimates with a quantitative uplift model.
Proposes a conceptual framework where only verified users can access dual-use model outputs, arguing this reduces both over-refusals and under-refusals while giving regulators targeted policy options.
A systematic review of risks in child-LLM interaction that proposes a protection framework with child-specific content safety, prompt-injection and jailbreak controls, and measurable evaluation targets.
Shows that fine-tuning via a jailbreak-tuning attack strips safety guardrails from DeepSeek R1 and fine-tunable GPT-4o, Claude 3 Haiku and Gemini models, arguing current safeguards are illusory and calling for evil-twin evaluation.
This perspective paper identifies four prompt-injection vulnerabilities in federated military LLMs and proposes a human-AI framework combining red/blue wargaming and joint policy verification as countermeasures.
Microsoft's lessons from red teaming over 100 generative AI products, covering threat models, automation, human expertise, and security risks of LLM systems, with practical recommendations for aligning red teaming with real-world risks.
Tests whether CVSS-style vulnerability metrics meaningfully distinguish adversarial attacks on LLMs across 56 attacks, finding little score variation and calling for LLM-tailored, more flexible metrics.
Tests five advanced LLMs across 24 EU languages, finding jailbreak vulnerability tied to language resourcing, and proposes EU AI Act policy measures such as mandatory multilingual safety assessments.
A survey categorising jailbreaks of LLMs and VLMs into seven types, reviewing defence strategies, and identifying research gaps for safer models.
Proposes PRISM, a framework for open-source foundation model safety using independent prompt and output moderation modules instead of value alignment, and finds open-source providers' acceptable use policies are looser than closed-source ones.
Proposes violet teaming, a framework combining red and blue teaming with ethics for responsible AI, and applies it to AI biosecurity risks and broader governance perspectives.
Surveys threats of LLM misuse for fraud, impersonation and malware, plus prevention measures and their vulnerabilities, presenting a taxonomy relating threats, mitigations and weaknesses.