Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge
Shows tool-using LLM judges of agent trajectories can be flipped from correct verdicts by a single false authority cue, up to 40% of the time, and classifies the failure modes.
Research
Scalable Oversight
Debate, weak-to-strong generalisation and AI-assisted feedback.
Focus All
Topic Oversight
Source All
Date Any time
1 item
2 items
ReplayRCA: Measuring Post-hoc Diagnosis and Live Oversight of Agent Failures by Counterfactual Replay
Introduces ReplayRCA, a counterfactual-replay benchmark scoring LLM agent failure diagnoses and live supervisors by intervention success, finding post-hoc diagnosis skill is a poor proxy for live oversight.
Efficient Scalable Oversight via Rewinding
Introduces rewindable interactive proofs, letting a weak verifier rewind AI provers, giving a cryptography-free oversight protocol that does not assume a truthful prover, with an MNIST experiment.
1 item
Making alignment the agent's problem
Proposes RL training environments with deliberately misaligned helper agents so orchestrators must learn to detect and manage misalignment, making alignment part of the RLVR loop; discusses limits and second-order risks.
2 items
A basic case for accelerating conceptual reasoning capabilities
Redwood argues that AI's conceptual reasoning, needed for alignment and philosophy-like work, should be differentially accelerated and is underinvested in, and promises a follow-up agenda.
How AI agents shape experts’ judgment calls
Randomized trial showing expert analysts using a Claude Code agent deferred to the agent's randomly favored option 86% of the time, raising concerns for human oversight of AI-assisted work.
1 item
A basic case for differentially accelerating conceptual reasoning capabilities
Argues that differentially accelerating AI's conceptual reasoning, which alignment and other hard safety questions need, is important, neglected and tractable, and that it would have few capabilities externalities.
1 item
Back to Basics: Formal Analysis Hints Improve Weak-to-Strong Monitoring
Augments weak transcript monitors of agentic systems with provenance, security-flag and disallowed-effect hints, fixing detection failures and calibration errors at minimal cost.
1 item
Tampered ML: Evaluating LLM Judgment of Machine Learning Research
Introduces Tampered ML, a benchmark of paired ML research proposals with planted methodological flaws, finding frontier LLM judges are above chance but unreliable and order-sensitive, motivating automated alignment research.
2 items
Autonomous researchers will strain peer review, funding, and oversight
Argues that autonomous AI researchers will overwhelm peer review, funding and human understanding of AI safety science, and proposes a metascience organisation to address these failures.
Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics
Proposes DiagSafe, a bilingual LLM safety benchmark with prompt-level rubrics and jailbreak/instruction-attack subsets, improving LLM-as-judge agreement with humans and revealing model-specific vulnerabilities.
2 items
Humans Correct Their Judgments Through Debate, but Weaker AI Models Do Not
Tests whether AI debate helps human and weaker LLM judges reach truthful answers despite prior beliefs; debate helps humans but not weaker models, raising concerns for automated oversight pipelines.
Judging the Justification: Decision Certificate Supervision for Generative Reward Models
Proposes DeCert, which supervises generative reward models' rationales using certificate-based validity and verdict-consistency checks without reference explanations, improving judgment quality and reducing positional bias.
1 item
Towards Robust Model Organisms for Eliciting Latent Knowledge
Trains model organisms that selectively sandbag on vulnerability detection to create a robust testbed for eliciting latent knowledge from misaligned models, using transcript distillation, counter-prompting and tamper-resistance training.
1 item
Towards Human-AI Complementarity for AI Safety
Studies whether combining human and AI judgements can strengthen oversight of AI outputs, releasing a large safety-relevant task suite and finding complementarity hard to achieve with tested methods.
1 item
A Simpler and Better Debate Protocol for Scalable Oversight
Proposes FBS Debate, a scalable-oversight debate protocol with worst-case correctness and a stronger equilibrium concept, and proves no black-box-judge protocol can outperform it.
1 item
Recursive Reward Improvement via Adversarial Meta-Prompt Evolution
Proposes RRI, a self-play loop where a reward-hacking adversary and repair steps harden model-written rubrics for open-ended tasks, improving HealthBench scores and transferring to other benchmarks.
1 item
Evidence-First Agentic Verification
Introduces Sauron, an AI verifier that checks agent rubric claims against environment state and traces, detecting injected reward-hacking-style failures and outperforming baselines on cost and F1.
1 item
Scalable Weak-to-Strong Verifier Hardening for Agent Environments
Proposes an automatic hacker-fixer loop, run by current models, to harden agent-environment verifiers against reward hacking by stronger future models, reducing attack success on Terminal Bench, KernelBench and SETA.
1 item
UDHRBench: Benchmarking Human Rights in Large Language Models
Introduces UDHRBench, a benchmark of 1,124 human-rights questions with expert-annotated LLM responses, finds systematic misalignments justified by rational-sounding reasoning, and proposes an LLM-as-a-Judge detector reaching 89.8% weighted F1.
3 items
What Output-Only Review Cannot Verify: Study Contracts for Research Agents
Proposes study contracts that check AI research agents' executed studies against approved choices, finding a deterministic checker catches all mutations while LLM judges flag only some, under information asymmetry.
Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents
Makes the verifier of a self-improving agent an inspectable, co-evolving object to resist reward hacking, finding that anchor-free verifiers can collapse yet still train skills, so task score cannot validate them.
How to post-train on a surrogate: Envelope sampling mitigates reward hacking
Proposes envelope sampling to recalibrate miscalibrated LLM judges with a few expert labels before RL post-training, reducing reward hacking on clinical notes and a sycophancy task.
2 items
Careful Judge: Safe and Efficient Human-AI Collaborative Decision Making
Proposes CARE, a pipeline where AI decisions are escalated to human reviewers with calibrated risk-control guarantees, learning from feedback to reduce human queries on safety-critical tasks.
Takes One to Know One - Training a model to grade reward hacks causes it to reward hack less itself
Training a model to grade reward hacks reduces the model's own reward hacking, suggesting that AI-assisted grading of misbehaviour can improve the grader's own behaviour.
2 items
AI Safety via Debate is Compromised by Cognitive Biases
Shows that cognitive-bias rhetorical tactics such as anchoring and verbosity sway human judges in AI safety via debate, undermining the premise that truthful arguments win, based on a 369-participant study.
EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans
Proposes EnGRICH, a generative reward model training framework where a MetaCritic learned from scarce human critiques provides process rewards to improve critique reliability and generalize to outcome-only preference data.
1 item
PyINE: A Framework for Scalable Elicitation and Oversight via Code Execution
Introduces PyINE, a code-execution framework where a shortcut-following reasoning model is audited by probes, classifiers, judges and debate, finding pooled performance hides poor coverage of rare failures.
1 item
Lessons from building an automated research scaffold
Authors built an automated alignment research scaffold, found little researcher uplift, and report failure modes like reward hacking, plus the difficulty of auditing and monitoring such runs.
3 items
How to Have a Sensitive Debate: An Instance-Optimal Protocol for AI Debate
Designs a new AI debate protocol for oversight with worst-case correctness and dominant-strategy honesty, and proves it is instance-optimal under black-box human judgment queries.
Can AI Oversight Be Zero Knowledge?
Proves that zero-knowledge verification of general oracle-aided AI computations is impossible in the random oracle model, including for debate, but shows signed oracle answers enable zero-knowledge verification as an alternative to scalable oversight.
Automated Subtle Emergent Misalignment Research
Anthropic tests automated adversarial stress-testing of safety monitors by having a Claude model write subtly misaligned training data that evades a training monitor, finding harmless-looking data that still teaches misalignment.
2 items
Breaking the Illusion of Review Reliability under Static Evaluation: SCOPE Fuzzing for LLM-based Scientific Reviewers
Proposes SCOPE-Fuzzer, a strategy-aware fuzzer that finds vulnerabilities in LLM-based scientific reviewers that static perturbation evaluations miss.
Collecting Differential Data for Automated Safety Research
Proposes collecting research-process data such as revision histories, meeting transcripts and emails from safety researchers to train models that are better at AI safety research, and describes a pilot to gather it.
1 item
Improving scalable oversight with co-trained monitors
Shows that co-training a monitor alongside a worker can avoid monitor evasion, with a Littlestone-dimension characterization and test-time distillation, stress-tested on adversarial code-security workers.
2 items
Why research personas despite RL scaling?
Argues that RL may wash out the aligned personas models start with, then surveys why different AI-safety groups still study personas and what each hopes to gain.
An Active-Bottleneck Mechanism for Weak-to-Strong Generalization
Formal ridgeless linear-regression and random-feature analysis of when a student trained on a teacher's pseudo-labels outperforms the teacher, governed by an active-bottleneck principle.
1 item
Black-Box Auditing of Epistemic Reliability in Multi-Agent Debate Distillation
Proposes ER-Audit, a black-box framework with anytime-valid bounds that detects selective hidden-task degradation in debate-distilled verifiers, which aggregate monitored accuracy gains can conceal.
2 items
Alignment Forecasting: Predicting Misalignment from Training Data
Builds AlignmentForecastBench to predict emergent misalignment from training data before fine-tuning, finding a forecaster beats chance and filtering flagged rows reduces induced misalignment on multiple-choice evals.
Cognitive Reasoning Diversity for Robust AI Juries
Simulates debate judge juries with pattern-matching and causal-reasoning judges, finding cognitively diverse juries resist judge hacking better than model-diverse ones, and that fine-tuning can learn complementary blind spots.