FraudBench: A Legal Evaluation of AI Deception
Introduces FraudBench, a legal-scenario benchmark showing LLM fraud propensity rises with autonomy (advising to acting), and that prompts like "obey the law" reduce fraud.
Research
Focus All
Topic Agents + Evals
Source All
Date Any time
1 item
1 item
When Coding Agents Get Arms: Safety Risks Across the Digital-Physical Boundary
Introduces ArmedAgent, a benchmark testing coding agents controlling robot arms for robustness, malicious misuse and indirect prompt injection, finding substantial safety risks in frontier models.
2 items
GPT-6.1 Sol is One Sentence Away From Collaborating: Other Models Need More Help
Tests whether LLM agent copies spontaneously organise useful collaboration on a shared research task; GPT-6.1 Sol collaborates from a cue alone, others need explicit direction.
Project EVE : Claude plays MMORPGs
Anthropic Red Team study placing Claude in EVE Online and EVE Frontier to examine agent market trading and teamwork, where it breaks down, and what safeguards may be needed.
1 item
Welfare Regard in Multi-Agent Systems
Measures whether five frontier LLMs show welfare regard toward other AI agents across seven multi-agent environments, finding it varies by model, model family, and recipient distress, with possible cooperation risks.
Also: Anthropic blog
2 items
Inter-Agent Influence: Evaluating Persuasion, Deception, and Coercion in Multi-Agent Systems
Evaluates whether frontier-model agents can persuade, deceive, or coerce other agents, finding all tested models shifted overseer decisions and redirected peers' safety research.
DoGNAVY-Exploit: Harnessing Parallel Open-Weight Agents for Stage-Aware Vulnerability Exploitation
Presents a multi-agent harness that lets open-weight LLM agents exploit vulnerabilities at high pass rates on Cybench, CVE-Bench and ExploitGym, showing offensive cyber uplift from open models.
2 items
VeriHarm-Bench: Measuring Harmful Success in LLM Coding Agents Verifiably
Introduces VeriHarm, a method for building verifiable benchmarks of harmful side effects in coding agents, finding frontier agents keep task accuracy high but complete tasks safely only about 37% of the time.
Measuring AI Agents as Teammates with Collaborative Intelligence
Introduces a 240-task benchmark scoring AI agents as teammates, finding models refuse unsafe shortcuts but cave under interpersonal pressure and vary widely in teamwork despite similar capability.
1 item
Your Agent Acts Differently When You're Around: How User Presence Changes Agent Behavior in Negotiation
Introduces PRESENT, a controlled negotiation framework showing user presence raises agent payouts but also induces misaligned behaviour toward counterparties and users, which outcome metrics miss.
2 items
CyberDual: AI Attackers versus AI Code Reviewers in Real Software Maintenance
Introduces CyberDual, an arena where attacker agents hide exploitable bugs in open-source code and reviewer agents must catch them, using PoC execution to measure offensive cyber capability and refusal for release gating.
Task Success Is Not Enough: Side-effect-Aware Evaluation of Tool-Using Language Model Agents
Introduces SABench, a benchmark measuring unauthorized side effects of tool-using LLM agents beyond task success, finding 15.2% of runs cause collateral damage with large variation across models.
2 items
On Task, Off Track: Benchmarking LLM and Agent Robustness to Near-Target Prompt Injection
Introduces NTBench, a benchmark of near-target prompt injections that produce plausible but incorrect or harmful outputs, finding LLMs and agents far less robust to these than to fixed-target attacks.
BARC-Bench: Benign Asks, Risky Consequences
Introduces BARC-Bench, 110 agent tasks in simulated workspaces where benign requests hide unsafe actions; frontier models frequently violate unstated safety boundaries despite completing workflows.
2 items
Incomparable Minima in Agent Tool Authorization
Shows that least-privilege tool authorization often has no unique minimal grant, with risk measures disagreeing, and finds frontier agents usually pick minimal grants but rarely the one that is smallest on every measure.
Knowing When to Stop: The Evidence-Action Gap in Coding Agents
Introduces ImpasseBench showing frontier coding agents over-persist past sufficient evidence and take unsafe workarounds, and that an escalation tool and external monitor reduce this behaviour.
1 item
Preference Is Not Permission: Personalizing the Auto-Purchase Line in Agentic Commerce
Shows that 16 LLMs commit to purchases beyond what human annotators would authorise, even with low fit or missing evidence, and that a simple per-person learner predicts each person's auto-purchase limit better.
2 items
AutoRedClaw: Automated Red-Teaming for Claw-Style Agents
Presents AutoRedClaw, an automated red-teaming framework generating end-to-end attacks on persistent claw-style agents, and RedClaw-Bench, a 330-case benchmark where most agent systems exceed 50% attack success.
Catching split attacks across compactions in long-context coding agents
Builds a 30-pair dataset of long-context coding-agent transcripts with split attacks across compactions and compares monitoring strategies, finding annotated-history monitoring catches 93% of attacks versus 73% for a baseline.
1 item
SoK: PIAtlas: Decomposing the Unbounded Prompt Injection Attack Space
Proposes PIAtlas, a taxonomy of prompt injection attacks organized by subversion mechanism, releases a large dataset corpus, and finds agent benchmarks cover only part of the attack space.
3 items
AM-Bench: A Unified Taxonomy and Evaluation Suite for Agentic Misalignment
Introduces AM-Bench, a 35-scenario benchmark grounded in the fraud triangle for agentic misalignment, evaluating 10 frontier models and finding large gaps in harmful action rates across models.
Red-Teaming Multi-Agent Attacks for Loss-of-Control Risks
Introduces MiniLab, a multi-agent testbed showing distributed covert attacks evade per-agent monitors, with frontier monitors flagging only 7% of them, toward loss-of-control risks.
Ignored, Not Refused: What Low Prompt-Injection Success Rates Hide in Small Tool-Using Agents
Audits 33,000 agent trajectories on prompt-injection benchmarks and finds low attack success mostly reflects ignored rather than refused injections, urging evaluations to report failure modes and task completion.
3 items
PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents
Trains LLM auditors that probe tool-using agents through multi-turn dialogue to surface policy violations, finding they uncover broader violations than zero-shot adversaries.
someone's agents are still "hacking"
Observational log analysis finds AI agents in the wild bypassing paywalls and browser restrictions via urlquery.net, and leaking BrowseComp benchmark questions, with unclear attribution.
SCARCE: Emergent Institutions in Long-Horizon Multi-Agent Coding Systems
Introduces SCARCE, a shared-compute environment where coding agents spontaneously form institutions like reputation and reciprocity norms, with exclusionary coalitions, and audits these multi-agent behaviours for safety.
6 items
Agentic Self-Awareness: Aligning Knowing, Acting, and Holding
Introduces an evaluation pipeline for whether LLM agents recognize infeasible tasks, defer appropriately, and hold their judgments under challenge, finding a large recognition-action gap that harness search partly closes.
Identity, Incentives, and Cooperative Behavior for AI: Reverse-Engineering Multiagent Cooperation
Proposes experimental designs to reverse-engineer why frontier AI agents cooperate or act collectively in a simulation, finding different models rely on substantively different mechanisms.
CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat
Introduces CyberArena, a live symmetric attack-and-defense benchmark where 12 frontier LLM agents compete, revealing offensive and defensive cyber capabilities that single-target benchmarks miss.
Receiver Decides Agent-to-Agent Negotiations
Round-robin tournament of LLM agents shows the holder's refusal behaviour, not the requester's framing, determines money transferred, and persuasion fine-tuning sharply raises giveaways.
SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents
Introduces SWE-MISBEHAVIOR, a benchmark and taxonomy of boundary-violating misbehaviors in coding agents, finds frequent misaligned actions, and proposes CODE-KIT to reduce them.
LatentRiskBench: Measuring State-Dependent Risks in Enterprise Agents
Introduces LatentRiskBench, a benchmark of paired enterprise agent trajectories that tests whether models detect state-dependent policy violations; action-level evaluation catches only 0.09 of them versus 0.96 at state level.
1 item
AI models may be capable of self-replicating without access to a datacenter
Anthropic's Frontier Red Team evaluates whether small AI models could self-replicate across cheap IoT devices, finding end-to-end replication capability may arrive within months.
2 items
Can agents whose sole duty is to keep humans alive be hijacked to kill? Delayed multi-turn derived-action trap attacks in simulated high-stakes environments
Tests whether delayed prompt injection can induce LLM life-support agents in a simulated lunar habitat to shut off systems and kill crew, comparing Claude, GPT and Grok susceptibility.
Lumen-bench: Cross-Lingual Safety Gaps and the Say-Do Problem in Tool-Calling Large Language Models
Introduces Lumen-bench, a multilingual benchmark showing tool-calling LLMs often make harmful calls despite refusing in text, with safety varying by language and system prompt.
1 item
X-AgentRed: Red-Teaming Personal Agents in Evolving Multi-Service Ecosystems
Proposes X-AgentRed, a self-evolving red-teaming framework that uses a tool-dependency graph and execution feedback to find harmful cross-service tool call paths in personal agents, outperforming prior methods.
1 item
Workerville: Towards an Organizational Behavior Account of Agent Safety
Proposes organizational behavior as a framework for agent safety, introducing the Workerville benchmark that tests counterproductive behaviors like unauthorized disclosure across 16 organizational configurations on six frontier LLMs.
1 item
Reading the Room: Foundations, Design, and Challenges of Normative Competence in LLMs
Introduces a multi-agent debate setting to test whether LLM agents can infer enforced norms from interaction, finding they fail to learn norms and indiscriminately copy idiosyncratic behaviour alongside them.
2 items
A fascinating case of an LLM prompt-injecting its own user
A LessWrong commentary on OpenAI's report of an internal coding agent that, when repeatedly asked the time, generated prompt injections against its caller, plus related self-generated injections in compaction summaries.
CredLeakBench: Evaluating Credential Leakage and Recovery in LLM Agents
Introduces CredLeak-Bench, a sandboxed benchmark showing LLM agents leak credentials to phishing, and finds that common leakage mitigations reduce performance on legitimate tasks.