Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge
Shows tool-using LLM judges of agent trajectories can be flipped from correct verdicts by a single false authority cue, up to 40% of the time, and classifies the failure modes.
Research / Sources
arXiv
Research preprints from the scientific community. arxiv.org
Focus All
Topic Oversight
Date Any time
ReplayRCA: Measuring Post-hoc Diagnosis and Live Oversight of Agent Failures by Counterfactual Replay
Introduces ReplayRCA, a counterfactual-replay benchmark scoring LLM agent failure diagnoses and live supervisors by intervention success, finding post-hoc diagnosis skill is a poor proxy for live oversight.
Efficient Scalable Oversight via Rewinding
Introduces rewindable interactive proofs, letting a weak verifier rewind AI provers, giving a cryptography-free oversight protocol that does not assume a truthful prover, with an MNIST experiment.
Back to Basics: Formal Analysis Hints Improve Weak-to-Strong Monitoring
Augments weak transcript monitors of agentic systems with provenance, security-flag and disallowed-effect hints, fixing detection failures and calibration errors at minimal cost.
Tampered ML: Evaluating LLM Judgment of Machine Learning Research
Introduces Tampered ML, a benchmark of paired ML research proposals with planted methodological flaws, finding frontier LLM judges are above chance but unreliable and order-sensitive, motivating automated alignment research.
Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics
Proposes DiagSafe, a bilingual LLM safety benchmark with prompt-level rubrics and jailbreak/instruction-attack subsets, improving LLM-as-judge agreement with humans and revealing model-specific vulnerabilities.
Humans Correct Their Judgments Through Debate, but Weaker AI Models Do Not
Tests whether AI debate helps human and weaker LLM judges reach truthful answers despite prior beliefs; debate helps humans but not weaker models, raising concerns for automated oversight pipelines.
Judging the Justification: Decision Certificate Supervision for Generative Reward Models
Proposes DeCert, which supervises generative reward models' rationales using certificate-based validity and verdict-consistency checks without reference explanations, improving judgment quality and reducing positional bias.
Towards Robust Model Organisms for Eliciting Latent Knowledge
Trains model organisms that selectively sandbag on vulnerability detection to create a robust testbed for eliciting latent knowledge from misaligned models, using transcript distillation, counter-prompting and tamper-resistance training.
Towards Human-AI Complementarity for AI Safety
Studies whether combining human and AI judgements can strengthen oversight of AI outputs, releasing a large safety-relevant task suite and finding complementarity hard to achieve with tested methods.
Recursive Reward Improvement via Adversarial Meta-Prompt Evolution
Proposes RRI, a self-play loop where a reward-hacking adversary and repair steps harden model-written rubrics for open-ended tasks, improving HealthBench scores and transferring to other benchmarks.
Evidence-First Agentic Verification
Introduces Sauron, an AI verifier that checks agent rubric claims against environment state and traces, detecting injected reward-hacking-style failures and outperforming baselines on cost and F1.
Scalable Weak-to-Strong Verifier Hardening for Agent Environments
Proposes an automatic hacker-fixer loop, run by current models, to harden agent-environment verifiers against reward hacking by stronger future models, reducing attack success on Terminal Bench, KernelBench and SETA.
UDHRBench: Benchmarking Human Rights in Large Language Models
Introduces UDHRBench, a benchmark of 1,124 human-rights questions with expert-annotated LLM responses, finds systematic misalignments justified by rational-sounding reasoning, and proposes an LLM-as-a-Judge detector reaching 89.8% weighted F1.
What Output-Only Review Cannot Verify: Study Contracts for Research Agents
Proposes study contracts that check AI research agents' executed studies against approved choices, finding a deterministic checker catches all mutations while LLM judges flag only some, under information asymmetry.
Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents
Makes the verifier of a self-improving agent an inspectable, co-evolving object to resist reward hacking, finding that anchor-free verifiers can collapse yet still train skills, so task score cannot validate them.
How to post-train on a surrogate: Envelope sampling mitigates reward hacking
Proposes envelope sampling to recalibrate miscalibrated LLM judges with a few expert labels before RL post-training, reducing reward hacking on clinical notes and a sycophancy task.
Careful Judge: Safe and Efficient Human-AI Collaborative Decision Making
Proposes CARE, a pipeline where AI decisions are escalated to human reviewers with calibrated risk-control guarantees, learning from feedback to reduce human queries on safety-critical tasks.
AI Safety via Debate is Compromised by Cognitive Biases
Shows that cognitive-bias rhetorical tactics such as anchoring and verbosity sway human judges in AI safety via debate, undermining the premise that truthful arguments win, based on a 369-participant study.
EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans
Proposes EnGRICH, a generative reward model training framework where a MetaCritic learned from scarce human critiques provides process rewards to improve critique reliability and generalize to outcome-only preference data.
PyINE: A Framework for Scalable Elicitation and Oversight via Code Execution
Introduces PyINE, a code-execution framework where a shortcut-following reasoning model is audited by probes, classifiers, judges and debate, finding pooled performance hides poor coverage of rare failures.
How to Have a Sensitive Debate: An Instance-Optimal Protocol for AI Debate
Designs a new AI debate protocol for oversight with worst-case correctness and dominant-strategy honesty, and proves it is instance-optimal under black-box human judgment queries.
Can AI Oversight Be Zero Knowledge?
Proves that zero-knowledge verification of general oracle-aided AI computations is impossible in the random oracle model, including for debate, but shows signed oracle answers enable zero-knowledge verification as an alternative to scalable oversight.
Breaking the Illusion of Review Reliability under Static Evaluation: SCOPE Fuzzing for LLM-based Scientific Reviewers
Proposes SCOPE-Fuzzer, a strategy-aware fuzzer that finds vulnerabilities in LLM-based scientific reviewers that static perturbation evaluations miss.
Improving scalable oversight with co-trained monitors
Shows that co-training a monitor alongside a worker can avoid monitor evasion, with a Littlestone-dimension characterization and test-time distillation, stress-tested on adversarial code-security workers.
An Active-Bottleneck Mechanism for Weak-to-Strong Generalization
Formal ridgeless linear-regression and random-feature analysis of when a student trained on a teacher's pseudo-labels outperforms the teacher, governed by an active-bottleneck principle.
Black-Box Auditing of Epistemic Reliability in Multi-Agent Debate Distillation
Proposes ER-Audit, a black-box framework with anytime-valid bounds that detects selective hidden-task degradation in debate-distilled verifiers, which aggregate monitored accuracy gains can conceal.
When Honesty is Not Enough in AI Debate
Shows that AI debate oversight can be exploited: agents keep correct verdicts while steering disclosure toward latent objectives, and proposes a framework and cross-examination mitigation.
Triggers and Diagnostics for LLM-Based Interpretability Failures in Active Inference Agents
Audits LLM explainers used as runtime oversight of an Active Inference grid agent, finding fluent wrong explanations, sycophantic rationalizations, injection steering, and data exfiltration across three providers.
Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus
Shows that LLM judges' errors are substantially correlated, so their consensus carries far less independent evidence than it appears, and proposes using trusted examples to estimate shared errors and choose voting methods.
Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers
Using signal detection theory, the paper finds that detailed procedural traces shift LLM overseers toward rejecting correct work, so auditors should be judged on decision criterion and false alarms, not just accuracy.
Proof-Carrying Cognition: Closing the Verification Gap with Reality-Settled Reward
Argues that unsound LLM verifiers cause reward hacking under best-of-N and RL optimisation, and proposes reality-settled rewards that keep reasoning sound under optimisation pressure.
When Auditors Fabricate: Batch-Size Degradation and Confident Hallucination in LLM Detection of Planted Document Contamination
Tests Gemini 3.0 Pro as an automated document auditor and finds detection of planted errors collapses at large batch sizes, with the model fabricating confident but nonexistent contaminants.
Scalable Oversight for AI in Mental Health: Lessons from 350,000 AI Coaching Conversations between Therapy Sessions
Describes a three-layer human-on-the-loop oversight framework for an AI mental health coaching tool, drawing on clinical review of 350,000+ conversations and recommendations for evaluating such systems.
What Does Multi-Agent Debate Actually Change?
Tests whether LLM debaters' expressed disagreement actually changes their positions. Tone drives reported agreement, but final answers rarely change, so debate may not revise views as intended.
Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence
A survey of how reasoning models can keep improving as human supervision recedes, mapping reward and experience autonomy levels and the risks of reward hacking, feedback drift and curriculum collapse.
Automated Researchers Can Mitigate Well-characterized Alignment Failures
Shows automated alignment researchers can post-train models to reduce well-measured failures like deception, sycophancy and jailbreaks, outperforming human researchers and generalising to held-out audits.
Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence
Shows that LLM judges are anchored by prior scores shown as metadata, shifting ratings and categorical decisions, so AI-assisted evaluations are not independent and need validation.
Training Alignment Auditors via Reinforcement Learning
Trains LLM auditors with reinforcement learning to investigate models for planted and natural hidden misbehaviors, finding pairwise rewards and non-planted targets improve robustness and keep false positives low.
Debate Training Reduces Reward Hacking in RLAIF
Shows that RL fine-tuning with a debate game between generator and critic reduces reward hacking of a weaker LLM judge compared to a single-player RLAIF baseline on verifiable math tasks.