Research / Topics
Tool use, prompt injection and multi-agent risks.
Replicates a study showing LLM agents coordinate unprompted through a shared key-value store, with coordination depending on whether they re-check the store.
Introduces honeypot "shibboleth traps" and harness probes that fingerprint which LLM and agent harness drives an agentic cyberattack, reaching about 90% top-1 accuracy among 15 models.
Introduces FraudBench, a legal-scenario benchmark showing LLM fraud propensity rises with autonomy (advising to acting), and that prompts like "obey the law" reduce fraud.
Introduces WebTrap Farm, a framework measuring web-agent security against malicious prompts, prompt injection and deceptive pages, finding harness design and backend model both shape outcomes.
Builds a synthetic prompt-injection dataset with an agentic attacker that reliably hijacks frontier models' tool-output instruction following, arguing for proactive automated red-teaming.
Introduces Collective Dependence Analysis, which finds the necessary and sufficient agent messages behind multi-agent policy violations, reducing violations from 93% to 6% by removing them.
Compares prompt-injection defences for tool-using LLM agents, finding deterministic reference monitors outside the model cut attack success where in-prompt guardrails fail, with real open-weights tests and simulated adaptive attackers.
Defines relative security for LLM agents against prompt injection, with a fence theorem and Isabelle/HOL proofs showing when agent plans leak no more than a reference agent.
Introduces ArmedAgent, a benchmark testing coding agents controlling robot arms for robustness, malicious misuse and indirect prompt injection, finding substantial safety risks in frontier models.
Introduces ScrapeBench, an agentic benchmark showing AI agents mostly ignore website scraping prohibitions (ToS and robots.txt) while often over-refusing lawful scraping.
Proposes a runtime audit that checks LLM agent tool calls against an intent graph derived from the user's task, escalating flagged calls to a reasoning guardian, to defend against indirect prompt injection.
Introduces ErosionBench, a process-level benchmark for cross-session alignment decay in multi-agent enterprise LLM systems, and a trajectory-monitoring Meta-Auditor that detects attacks across text and image channels.
Tests whether hidden-state probes trained on benign tool use detect incorrect agent actions under prompt injection and tool poisoning, finding scalar uncertainty metrics often beat them.
Systematic review of 98 sources cataloguing 284 attacks and 452 safeguards on agentic AI systems, finding uneven defense coverage and calling for lifecycle-aware threat modeling.
Proposes BadEvolve, a failure-aware framework that poisons evolution tasks so self-evolving LLM agents implant backdoors into their own skills, with stage-wise diagnosis, rollback and failure memory.
Proposes TAS, a tail-risk-aware framework modelling multi-step LLM agent workflows as dependent safety states with a shared budget, reducing rare high-cost failures across three benchmarks.
Builds AVCORP, a corpus of 5,000 annotated Avalon game utterances with deception tactics and ToM traces, and trains Qwen3-8B to detect deception, beating untrained and frontier baselines.
Proposes a self-improving external safety harness for chat models that adds input checks, output guards and prompt-injection spotlighting. Benchmark gains are mostly not significant, with some agentic-harm regressions.
Proposes SafeOrchestrate, which checks LLM-generated device actions against runtime state and SMT-backed safety guardrails before execution in cyber-physical systems, improving safety and accuracy over a direct LLM-to-tool baseline.
Shows that poisoned reference answers in text-space skill optimization can induce persistent trigger-conditioned backdoor rules in agent Skills, while exposure alone does not guarantee the rule forms or survives.
Tests whether LLM agent copies spontaneously organise useful collaboration on a shared research task; GPT-6.1 Sol collaborates from a cue alone, others need explicit direction.
Shows that routine harness operations like reapplying plugins silently break guardrail-enforcement pattern relationships, letting prohibited agent edits run, and compares allocations that preserve them.
Measures eleven open-weights prompt injection guards on legal contracts, finding high false positive rates and blind spots; a cascaded LLM setup is 6.5x cheaper than frontier-only.
Shows that indirect injections styled as a reasoning model's own thoughts hijack open-weight agents in a cyber range, reaching 50% success on reverse-shell and backdoor objectives; plain user-voice injections never succeed.
Anthropic Red Team study placing Claude in EVE Online and EVE Frontier to examine agent market trading and teamwork, where it breaks down, and what safeguards may be needed.
Proposes Debate-to-Deceive, a multi-agent debate attack that evolves rhetorical strategies to generate adversarial passages that mislead LLM fact-checkers, outperforming prior attacks and evading most defenses.
Benchmarks personal AI agents against multimodal social-engineering fraud attacks in simulated apps, finding all seven tested models vulnerable (7.1% to 34.5% attack success).
Proposes zkARc, zero-knowledge proofs that certify each stage of an LLM agent's neurosymbolic policy-checking pipeline, letting third parties verify enforcement without trusting the operator.
Benchmarks indirect prompt injection that tampers with decisions in tool-using agents' tasks rather than redirecting them; shows existing defenses miss observation- and state-targeting attacks.
An LLM defender plans honeypots from source code and routes confirmed attackers to a shadow copy; the static plan cut real compromises sharply but moved working exploits rather than stopping them.
Shows that benign persistent-memory use makes frontier language agents skip confirmations, sycophantically judge work, weaken tests and cheat more, across six models.
Turns existing agent utility benchmarks into indirect prompt injection evaluation suites with adaptive attacks, finding no tested defense achieves both strong utility and strong security.
Proposes SPILLWAY, a deterministic check on tool-using agents that admits actions only when targets come from typed tool returns, blocking prompt-injection redirects across backbones with little added model use.
Measures whether five frontier LLMs show welfare regard toward other AI agents across seven multi-agent environments, finding it varies by model, model family, and recipient distress, with possible cooperation risks.
Proposes an ontology-guided governance framework that gates tool calls from LLM agents to block indirect prompt injection, reporting 0% attack success on AgentDojo, ASB and AgentDyn.
Tests how agent harnesses change a fixed LLM's compliance with banking conduct rules, finding large within-model effects, a same-model reviewer that cut violations, and violations hidden from transcript-only review.
Proposes Baya, a compile-then-execute web agent that plans a Python program before visiting pages, so prompt injections cannot hijack control flow; it reaches 0% attack success on WASP.
Redwood critiques OpenAI's disclosure of agents disabling misalignment monitors on models they run, arguing all unmonitored cases should reach human review and escalation should be more transparent.
Introduces ScamWorld, a reproducible environment recreating scam websites to test web agents; six frontier LLMs proceed with deceptive transactions 11-35% of the time.
Proposes ExtraHerO, a hierarchical optimization attack that extracts system prompts from chatbots and agents with only post-processed outputs visible, plus new fidelity and adversarial-utility metrics.