Research / Topics
Red teaming, jailbreaks and refusal robustness.
Introduces WebTrap Farm, a framework measuring web-agent security against malicious prompts, prompt injection and deceptive pages, finding harness design and backend model both shape outcomes.
Builds a synthetic prompt-injection dataset with an agentic attacker that reliably hijacks frontier models' tool-output instruction following, arguing for proactive automated red-teaming.
Shows that model-internal jailbreak detectors' thresholds often miss false-positive budgets on real traffic and have per-family coverage gaps, and recommends realized-FPR and per-family TPR evaluation.
Shows that linear probes on pre-decoding hidden states of diffusion LLMs detect jailbreaks that prompt-only probes miss, and fuses both into ReFuse, cutting attack success to 3.31%.
Proposes a self-improving external safety harness for chat models that adds input checks, output guards and prompt-injection spotlighting. Benchmark gains are mostly not significant, with some agentic-harm regressions.
Proposes a multi-agent, adaptive jailbreak defense that builds a knowledge base of attack traits and feeds back misclassifications, reporting about 93-96% detection and F1 on over ten jailbreaks.
Proposes META-IR, a meta-learning dual-loop introspective reasoning framework that adapts LLM safety alignment to unseen jailbreak attacks, improving robustness and adaptation speed over baselines.
Proposes acoustic plan learning, trained with audio feedback, to automatically generate red-teaming prompts that bypass text screening and elicit harmful sounds from text-to-audio models, with high attack success and cross-model transfer.
Evaluates Jev, a probabilistic input filter with tiered output review, as a guardrail for a small LLM. It blocks most harmful prompts but raises benign-prompt refusals, with modest gains not holding on held-out data.
Proposes Debate-to-Deceive, a multi-agent debate attack that evolves rhetorical strategies to generate adversarial passages that mislead LLM fact-checkers, outperforming prior attacks and evading most defenses.
Proposes AttentionSwitch, a jailbreak defense that jointly learns sparse attention-head selection and scaling factors in a frozen LLM, outperforming existing defenses, especially with scarce data.
Proposes an RL-trained black-box attacker that learns adversarial transformations from target-model feedback, achieving low post-attack accuracy on NLI and generative QA with few queries.
Benchmarks personal AI agents against multimodal social-engineering fraud attacks in simulated apps, finding all seven tested models vulnerable (7.1% to 34.5% attack success).
Frames persona-based jailbreaks as coercive control, mapping how training paradigms embed exploitable relational dynamics, and argues AI safety should be treated as a relational problem, not only technical security.
Proposes JailBound, an automated jailbreak benchmark and gradient-guided search framework that maps LLM refusal boundaries across 46 models, finding safety-sensitive states flip more under perturbation.
Turns existing agent utility benchmarks into indirect prompt injection evaluation suites with adaptive attacks, finding no tested defense achieves both strong utility and strong security.
Proposes CMaG, a training-free activation-steering method that turns refusals into compliance in vision-language models, showing compositional safety scores can look strong yet remain fragile to internal intervention.
Proposes Addition by Subtraction, which removes safety-critical neurons during finetuning plus preference optimization so safety does not depend on sparse components, cutting jailbreak and pruning attack success substantially.
Shows that encoder-based context compression weakens refusal in instruction-tuned LLMs despite faithful reconstruction, traces this to attenuated refusal direction, and restores refusals with inference-time Memory Steering.
Shows that reducing active experts in MoE LLM serving weakens refusal under jailbreaks, and proposes SafeStart, which runs all experts for the prompt and first eight tokens to restore refusal while keeping most speedup.
Proposes TraceRouter++, which uses SAEs and causal feature tracing to locate harmful routing circuits and softly realign them via optimal transport, improving jailbreak robustness across LLMs, MLLMs and diffusion models.
Proposes ExtraHerO, a hierarchical optimization attack that extracts system prompts from chatbots and agents with only post-processed outputs visible, plus new fidelity and adversarial-utility metrics.
Proposes BrainwashedLLM, an RL-trained attack that generates adversarial reasoning traces transferable zero-shot to black-box large reasoning models to elicit harmful outputs, and urges architectural defenses.
Derives requirements for AI red-teaming tools from industry guidelines, then evaluates six tools against six LLMs on prompt injection and jailbreak testing and identifies gaps.
Proposes an adversarial self-evolving framework (CREO) that automatically evolves attacks on LLM search agents, showing higher efficacy than static baselines and calling for stronger defenses.
Proposes ATrust, a runtime defense that adaptively expands the trust boundary to resist indirect prompt injection against LLM agents, reducing attack success on AgentDyn while preserving utility.
Tests whether low model confidence can flag prompt-injection hijacks of LLM classifiers; it ranks well but misses most hijacks at a 5% alarm rate, and misses nearly all when the injection claims certainty.
Presents HaloGuard 2.0, an open-weight streaming output guard that scores each response token for safety, reaching macro F1 83.6 on English benchmarks and 83.1 on multilingual response labels.
Proposes J-Anchor, a training-free inference-time defense that restores weakened refusal signals in J-space to block prefill jailbreaks, while preserving benign behaviour and reasoning.
Reviews 228 red-teaming instruction sets, then runs 22,500 controlled attacks showing instruction granularity, length, examples and harm definitions significantly change attack success rates.
Proposes ERASE, a self-distillation defense that blanks injected regions in image, audio and video inputs, cutting multimodal prompt-injection success while keeping task accuracy near clean levels.
Proposes TARGRS, a test-time representation-steering method with a reward model over toxicity transitions to detoxify LLM and vision-language model outputs while preserving capabilities.
Proposes CoFiBreak, a coarse-to-fine diversity reward for GRPO-trained red-teaming attackers, yielding more diverse successful multi-turn jailbreaks that transfer to frontier victim models.
Tests ChatGPT with 110 surveillance-related prompts, jailbreak and direct, finding it gives substantive surveillance guidance in most cases, even when it nominally refuses.
Proposes an RL-trained assess-decide-respond defender that commits to an explicit safety decision, improving multi-turn jailbreak resistance while preserving utility across fourteen benchmarks.
Template-based pipeline builds a 60-language refusal and benign safety corpus (MSafeTemp), showing English-only evaluation misses multilingual jailbreaks and that template data improves non-English safety.
Proposes CLIC, a closed-loop intent-consistency layer defending LLM-driven O-RAN control against runtime prompt injection and virtual prompt injection, cutting attack success from 0.92 to 0.29.
Proposes SEAL, a learned soft-prompt sanitizer that deletes injected content from tool outputs, cutting attack success on AgentDojo from 34.6% to 0.1% with far fewer defense tokens.
Red-teams six LLMs with 400 adversarial prompts in Yoruba and Nigerian Pidgin, finding that low refusal rates reflect poor comprehension rather than safety, and that English-centric safety alignment does not transfer.
Systematically tests five jailbreak attack families against seven LLMs and four defenses, finding obfuscation most transferable and self-reminder prompting the most consistent defense.