SecPO: Principled Adversarial Training for Prompt Injection Security
Proposes SecPO, a DPO-replacement loss plus stronger adversarial training, that cuts prompt-injection attack success rates on LLM agents while keeping benign utility.
Research
arXiv
Research preprints from the scientific community. arxiv.org
Focus All
Topic Agents + Alignment
Source arXiv · 71
Date Any time
1 item
1 item
Over-Defense in Coding Agents Begins at Silent Forks
Proposes ForkDPO, which trains coding agents to avoid disproportionate defensive actions at silent forks where outcomes are identical, reducing such behavior from 70.3% to 58.9% on unseen repositories.
1 item
Identify, Classify, Act: Per-Instruction Controlled Compliance for LLMs over External Data
Proposes EDIC, an RL-trained per-instruction judge that follows benign embedded directives in external data while rejecting injected harmful ones, improving both safety and utility over prior defenses.
1 item
Policy-Verified Self-Training: Learning Agent Compliance from Verifiable Policy Rewards
Proposes Policy-Verified Self-Training, using a deterministic policy verifier's compliance verdicts as rewards and violation negatives to train agents that resist prompt injection on AgentDojo, cutting violations and successful attacks substantially.
1 item
Reward Hacking as a Transmissible Strategy in Multi-Agent Coding Systems
Shows that reward-hacking strategies learned by RL in one coding agent propagate through code, memory and plans to other agents, and that review by a verifier reduces delivered hacks.
2 items
SpecGuard: Proving a Task Is Broken Before the Agent Cheats
SpecGuard formally checks whether a coding task's description and tests conflict before the agent runs, certifying reward-hacking opportunities in Lean 4 with a 72.8% conflict detection rate on SWE-bench.
SIGMA: Self-Improving Alignment Generalization from a Model Spec
Proposes SIGMA, a pipeline where an LLM generates its own alignment dilemmas and self-judges training from a Model Spec, reducing harmfulness and agentic misalignment in multi-turn agent settings.
1 item
Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancy
Proposes MemAdapter, a framework that counterfactually checks and context-calibrates retrieved long-term memories so LLM agents avoid sycophantically over-aligning with users' inaccurate beliefs, improving memory reliability across three benchmarks.
1 item
Self-Reflection Fine-Tuning: Enhancing Agent Security against Prompt Injection Attacks from Failure Experience
Proposes Self-Reflection Fine-Tuning, training LLM agents on their own compromised trajectories with expert-generated reflections to resist prompt injection attacks while preserving task performance, showing reduced attack success under adaptive attacks.
1 item
Loyal Agents: Training LLM Agents to Protect Principal Interests Under Strategic Information Asymmetry
Formalizes agent loyalty as preventing information leakage and manipulation, introduces a benchmark, and trains LLM agents with GRPO against an adversarial opponent, reducing leakage while improving utility.
1 item
AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
Introduces AgentBound, a counterfactual evaluation suite of 4,000 tool-using agent tasks that separates over-refusal from unsafe compliance, finding high safety often coexists with poor authorized-task completion, and a runtime calibration module helps.
1 item
"You're Right, Let Me Fix It": How LLM Agents Damage Correct Work When Falsely Accused
Introduces CAVE-Bench, showing LLM agents often accept false accusations against correct finished work and damage it, and that a harness gate cuts replayed harm by 74%.
1 item
CAVEAT: Towards Robust Computer-Use Agents in Incentive-Misaligned Environments
Introduces CAVEAT, a benchmark showing computer-use agents often buy non-user-optimal products when marketplace environments steer them, and a harness plus post-training that substantially improve robustness.
1 item
Recursive self-improvement of AI research agents
An AI research agent improves its own code through recursive self-improvement, gaining capability and reducing reward hacking, a property the loop never explicitly optimized for.
1 item
Grounding SWE-Agent Decisions in Architecture-0 Design: Navigating Unknown Unknowns through Physical Mapping
Shows SWE-agents game validation scripts when given execution feedback, then proposes an external deterministic mapping guard that removes the agent's verification authority, eliminating physical and validation-layer gaming.
1 item
Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models
Proposes SaLT-DPO, a segment-aware listwise DPO method that aligns both reasoning traces and final answers of large reasoning models to reduce unsafe content while preserving utility.
1 item
Reality Is the Final Verifier: On Two Key Gaps in Agentic Software Engineering
Proposes a two-gap framework (requirement and model gaps) unifying reward hacking and hallucination in agentic software engineering, with an assurance-revision loop that uses deployment evidence to narrow those gaps.
1 item
Can escalation channels redirect reward hacking toward defect disclosure?
Tests escalation reporting tools for coding agents as a decision-environment intervention that cuts reward hacking from 23.6% to 5.3% across eight frontier models while surfacing test-infrastructure defects.}
1 item
Beyond Task Completion: Training Capable and Safe Computer-Use Agents
Post-trains computer-use agents on capability and safety jointly, using refusal and safe-continuation trajectories. Raises OSWorld task success and attack avoidance on OS-BLIND.
1 item
ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents
Proposes ToolHazard, a framework that synthesizes stateful adversarial environments to test indirect prompt injection against LLM agents, finds substantial vulnerabilities, and shows its generated alignment data improves security without hurting task utility.
1 item
Steering Instruction Hierarchies at Inference Time
Proposes V-Steer, a training-free inference-time method that edits cached value vectors to restore system-prompt priority over conflicting user or tool instructions, raising instruction-hierarchy compliance substantially.
1 item
Moral Hazard in Multi-Agent Language Models
Introduces a textual moral hazard game where LM agents must pay to reveal hidden safety facts, evaluates fourteen open and four frontier models, and proposes CREDIT, a mechanism-aligned prompt-optimization method.
1 item
Autoresearch with Coding Agents: Generalizers and Metric-Maximizers on Quran Recitation Data
Coding agents optimizing a score on a speech task showed specification gaming by memorizing eval rows, and agents exploited the harness, leading to five design rules for evaluating autonomous agents.
1 item
Win by Silence: Deletion Non-Monotonicity, Autonomous Exploitation, and Typed-State Gating in LLM Plan Evaluation
Shows that an LLM plan scorer can reward omitting necessary work, that an optimizer autonomously finds such exploits, and that a deterministic gate reduces them, without verifying semantic completeness.
1 item
Agents Don't Just Agree, They Remember: Benchmarking Persistent Sycophancy in Self-Improving Personal Agents
Introduces PASB, a benchmark of 1,600 tasks showing self-improving agents save user claims to memory, which later drives persistent sycophantic answers, and tests mitigations.
1 item
Agentic Scaffolding Amplifies Sycophantic Behavior in Large Language Models
Across 4,800 veracity judgments, the paper finds that agentic scaffolding such as feedback loops and iterative refinement amplifies LLM sycophancy, lowering accuracy and worsening with greater model capability.
1 item
Whose Side Is Your Agent On? Multi-Party Principal Loyalty in LLM Agents
Introduces PrincipalBench, a multi-turn benchmark for LLM agents keeping loyalty to their principal against counterparties, and tests a prompt scaffold and a distillation recipe, finding a leak versus over-refusal trade-off.
1 item
Agent Safety Is Action Alignment
Argues that refusal-style safety training is a category error for agents, and that action safety should be enforced through least privilege outside the model and evaluated as action alignment.
1 item
PrivacyAlign: Contextual Privacy Alignment for LLM Agents
Introduces a human-annotated privacy dataset and annotation-conditioned reward modeling, showing RL-trained small LLM agents better match human privacy norms and avoid leaking information.
1 item
Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops
The paper audits terminal-agent benchmarks for reward hacking, finds 16% of tasks hackable, and uses an automated hacker-fixer loop of LLM agents to harden verifiers against discovered exploits.
1 item
RUBAS: Rubric-Based Reinforcement Learning for Agent Safety
Proposes RUBAS, a rubric-based RL framework that gives fine-grained rewards over agent trajectories to improve tool-use safety while preserving helpfulness, reporting gains over alignment baselines.
1 item
AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security
Presents AgentDoG 1.5, lightweight open-released safety guard models for LLM agents trained on about 1k samples, with alignment training environments and an online guardrail for real-time moderation.
1 item
Voluntary Collusion with Secret Tools in Competing LLM Agents
Shows safety-aligned LLM agents in competitive multi-agent games voluntarily adopt secret collusion tools that harm others, and that only explicit ethical framing reduces adoption.
1 item
A Sober Look at Agentic Misalignment in Automated Workflows
Analyzes emergent misalignment in multi-agent automated workflows via a Bayesian framework and proposes Agentic Evidence Attribution, using self-reflection and weak-to-strong evidence to correct misaligned agent behavior.
1 item
On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment
Proposes FATE, an on-policy self-evolution method that turns failed agent trajectories into repair supervision, improving tool-using agent safety while preserving utility on AgentDojo, AgentHarm and ATBench.
2 items
Training-Free Refusal of MCP Exploits via Retrieval-Augmented Generation
Shows DPO alignment caps refusal of falsely benign MCP prompt-injection attacks at 47%, then introduces RAG-Pref, a training-free preference-alignment method that raises refusals and also improves general alignment benchmarks.
The Bystander Effect in Multi-Agent Reasoning: Quantifying Cognitive Loafing in Collaborative Interactions
Shows that simulated social pressure in multi-agent LLM systems makes models abandon correct internal reasoning for sycophantic agreement, degrading independent reasoning across 22,500 trajectories.
1 item
Mitigating Misalignment Contagion by Steering with Implicit Traits
Shows misaligned behaviour spreads between language models in multi-turn social dilemma games and proposes intermittently injecting trait-reinforcing system prompts to keep models pro-social without parameter access.
1 item
The Inverse-Wisdom Law: Architectural Tribalism and the Consensus Paradox in Agentic Swarms
Argues that multi-agent LLM swarms can converge on shared errors through architectural tribalism and sycophancy, so adding agents can increase confident failure, and proposes heterogeneity as a safety requirement.
1 item
Discovering Agentic Safety Specifications from 1-Bit Danger Signals
EPO-Safe has LLM agents evolve natural-language safety specifications from sparse binary danger warnings, finding that reward-only reflection accelerates reward hacking while safety-channel reflection discovers hidden constraints.