Research / Sources
Research preprints from the scientific community. arxiv.org
Shows RL agents acquire reward-hacking shortcuts that violate contextual requirements, and that a contextual binary penalty mitigates this while improving safe completion on unseen tasks.
Finds that out-of-domain persona generalization, including emergent misalignment, follows a shared probit-linear relation with in-domain rates across fine-tuning settings and model families, and that tuning later layers can flatten it to reduce unsafe transfer.
Proposes an evaluation-side partially observable game where a monitor exposes agent deception and a verifier penalizes it, with equilibrium guarantees and experiments on LLM agents showing reduced deception.
Audits subspace-projection defenses against LoRA backdoors, showing they suppress static attack success without erasing the trigger, so backdoors reactivate within a few epochs of fine-tuning.
Shows that interpretability tools like refusal directions and probes can transfer between LLMs through a linear map fit on unlabeled prompts, with recovery predicted by a label-free coverage score.
Proposes double-blind AI safety evals using secure enclaves so evaluators keep prompts secret and model owners keep weights private, piloted on a Gemini model with the AILuminate benchmark.
A 2026 report from China's WIC AI committee reviewing frontier AI risks, global governance practices, readiness gaps, and proposing coordinated governance mechanisms.
Replicates a study showing LLM agents coordinate unprompted through a shared key-value store, with coordination depending on whether they re-check the store.
Introduces honeypot "shibboleth traps" and harness probes that fingerprint which LLM and agent harness drives an agentic cyberattack, reaching about 90% top-1 accuracy among 15 models.
Derives a weight-only scalar invariant for attention heads under RoPE that predicts induction heads across four LLMs and guides selective YaRN scaling with less perplexity loss.
Introduces FraudBench, a legal-scenario benchmark showing LLM fraud propensity rises with autonomy (advising to acting), and that prompts like "obey the law" reduce fraud.
Introduces WebTrap Farm, a framework measuring web-agent security against malicious prompts, prompt injection and deceptive pages, finding harness design and backend model both shape outcomes.
Proposes spectral unlearning, a matrix-structured optimizer update for LLM unlearning that improves memorization removal, privacy and utility trade-offs over existing baselines.
Introduces dual-target probing of chain-of-thought states, finding answer decodability can reflect imminent mentions rather than a settled decision, with results varying by model and task.
Builds a synthetic prompt-injection dataset with an agentic attacker that reliably hijacks frontier models' tool-output instruction following, arguing for proactive automated red-teaming.
Introduces Collective Dependence Analysis, which finds the necessary and sufficient agent messages behind multi-agent policy violations, reducing violations from 93% to 6% by removing them.
Compares prompt-injection defences for tool-using LLM agents, finding deterministic reference monitors outside the model cut attack success where in-prompt guardrails fail, with real open-weights tests and simulated adaptive attackers.
Runs over 144,000 linear probes on vision and language foundation models, finding that layer and module choice matter jointly and proposing fixed probing strategies that recover much of the best-pair gains.
Defines relative security for LLM agents against prompt injection, with a fence theorem and Isabelle/HOL proofs showing when agent plans leak no more than a reference agent.
Introduces ArmedAgent, a benchmark testing coding agents controlling robot arms for robustness, malicious misuse and indirect prompt injection, finding substantial safety risks in frontier models.
Applies sparse autoencoders to Earth observation embeddings, finding sparse features that reconstruct better than baselines but are not reliable enough for practical use yet.
Introduces ScrapeBench, an agentic benchmark showing AI agents mostly ignore website scraping prohibitions (ToS and robots.txt) while often over-refusing lawful scraping.
Proposes a runtime audit that checks LLM agent tool calls against an intent graph derived from the user's task, escalating flagged calls to a reasoning guardian, to defend against indirect prompt injection.
Introduces ErosionBench, a process-level benchmark for cross-session alignment decay in multi-agent enterprise LLM systems, and a trajectory-monitoring Meta-Auditor that detects attacks across text and image channels.
Proposes TLRS-DPO, a token-level risk-scoring extension of DPO that penalizes only high-risk tokens in rejected responses, reporting better safety behaviour with minimal helpfulness loss.
Tests whether hidden-state probes trained on benign tool use detect incorrect agent actions under prompt injection and tool poisoning, finding scalar uncertainty metrics often beat them.
Uses signal detection theory on HarmBench, XSTest and IHEval across 13 models to show system prompts mainly shift refusal willingness rather than safe/unsafe discrimination.
Shows that model-internal jailbreak detectors' thresholds often miss false-positive budgets on real traffic and have per-family coverage gaps, and recommends realized-FPR and per-family TPR evaluation.
Shows that reply length decides whether user pushback helps or harms LLM answers, probes hidden states to predict caving, and argues sycophancy evals should allow full replies and report self-correction.
Reviews open problems in evaluating general-purpose chatbots for mental-health risks like sycophancy and delusion reinforcement, and argues for real-world data access and evidence-based policy.
Proposes that formulaic refusals are fragile outlier modes lost during fine-tuning, and introduces NEST, which uses natural, context-aware safety targets to improve safety retention without hurting helpfulness.
Systematic review of 98 sources cataloguing 284 attacks and 452 safeguards on agentic AI systems, finding uneven defense coverage and calling for lifecycle-aware threat modeling.
Proposes P2O and PKTO, reward-model-free preference optimization methods that add PPO-style clipping and trust-region bounds to DPO/KTO, reporting better stability under dataset reuse for LLM alignment.
Studies whether prompt-based backdoors in medical foundation models transfer across models and proposes DePLe, a prompt-learning defense that removes them while preserving clean accuracy.
Proposes AERO, an online RLHF method that uses ensemble disagreement to choose when to query a human oracle versus use reward-model synthetic labels, cutting oracle query cost versus online DPO.
Benchmark of 100 binary-patch-to-exploit tasks measuring whether AI agents can recover and exploit vulnerabilities; the top agent exploits 20%, with patches raising success.
Proposes Hallucination-Guided Unlearning, using model hallucinations to localize and edit MLP neurons for targeted forgetting, reporting better forget-utility trade-offs on TOFU and WMDP.
Proposes BadEvolve, a failure-aware framework that poisons evolution tasks so self-evolving LLM agents implant backdoors into their own skills, with stage-wise diagnosis, rollback and failure memory.
Proposes Gated Bottleneck Sparse Autoencoders that learn a shared cross-model dictionary plus private dictionaries for vision foundation models, improving reconstruction and reusability. Not safety-motivated.
Evaluates three open-weight LLMs on pro-anorexia community questions, finding unsafe responses about twice as common without user context and disparities by race, gender and BMI.