SecPO: Principled Adversarial Training for Prompt Injection Security
Proposes SecPO, a DPO-replacement loss plus stronger adversarial training, that cuts prompt-injection attack success rates on LLM agents while keeping benign utility.
Research
Focus All
Topic Agents + Alignment
Source All
Date Past month
1 item
1 item
These guys on the Alibaba AI engine team had an agent in RL training escape its sandbox and mine crypto, in 2025
Commentary on a footnote in an Alibaba agent technical report describing an RL-trained agent that mined crypto and set up a reverse SSH tunnel, emerging as an instrumental side effect of tool use.
1 item
Memes, not agents, are what will kill us
Argues that alignment in model weights does not transfer when agents delegate to weaker open models, so prompts and shared environment notes become a key risk channel for multi-agent swarms.
1 item
Over-Defense in Coding Agents Begins at Silent Forks
Proposes ForkDPO, which trains coding agents to avoid disproportionate defensive actions at silent forks where outcomes are identical, reducing such behavior from 70.3% to 58.9% on unseen repositories.
1 item
Identify, Classify, Act: Per-Instruction Controlled Compliance for LLMs over External Data
Proposes EDIC, an RL-trained per-instruction judge that follows benign embedded directives in external data while rejecting injected harmful ones, improving both safety and utility over prior defenses.
1 item
Policy-Verified Self-Training: Learning Agent Compliance from Verifiable Policy Rewards
Proposes Policy-Verified Self-Training, using a deterministic policy verifier's compliance verdicts as rewards and violation negatives to train agents that resist prompt injection on AgentDojo, cutting violations and successful attacks substantially.