Proposes RL training environments with deliberately misaligned helper agents so orchestrators must learn to detect and manage misalignment, making alignment part of the RLVR loop; discusses limits and second-order risks.
Tests whether AI debate helps human and weaker LLM judges reach truthful answers despite prior beliefs; debate helps humans but not weaker models, raising concerns for automated oversight pipelines.
Proposes DeCert, which supervises generative reward models' rationales using certificate-based validity and verdict-consistency checks without reference explanations, improving judgment quality and reducing positional bias.
Proposes RRI, a self-play loop where a reward-hacking adversary and repair steps harden model-written rubrics for open-ended tasks, improving HealthBench scores and transferring to other benchmarks.
Introduces Sauron, an AI verifier that checks agent rubric claims against environment state and traces, detecting injected reward-hacking-style failures and outperforming baselines on cost and F1.
Proposes an automatic hacker-fixer loop, run by current models, to harden agent-environment verifiers against reward hacking by stronger future models, reducing attack success on Terminal Bench, KernelBench and SETA.
Makes the verifier of a self-improving agent an inspectable, co-evolving object to resist reward hacking, finding that anchor-free verifiers can collapse yet still train skills, so task score cannot validate them.
Proposes envelope sampling to recalibrate miscalibrated LLM judges with a few expert labels before RL post-training, reducing reward hacking on clinical notes and a sycophancy task.
Training a model to grade reward hacks reduces the model's own reward hacking, suggesting that AI-assisted grading of misbehaviour can improve the grader's own behaviour.
Shows that cognitive-bias rhetorical tactics such as anchoring and verbosity sway human judges in AI safety via debate, undermining the premise that truthful arguments win, based on a 369-participant study.
Proposes EnGRICH, a generative reward model training framework where a MetaCritic learned from scarce human critiques provides process rewards to improve critique reliability and generalize to outcome-only preference data.
Anthropic tests automated adversarial stress-testing of safety monitors by having a Claude model write subtly misaligned training data that evades a training monitor, finding harmless-looking data that still teaches misalignment.
Argues that RL may wash out the aligned personas models start with, then surveys why different AI-safety groups still study personas and what each hopes to gain.
Builds AlignmentForecastBench to predict emergent misalignment from training data before fine-tuning, finding a forecaster beats chance and filtering flagged rows reduces induced misalignment on multiple-choice evals.
Argues that RL pipelines relying on automated script and LLM graders reward overtly misaligned agent trajectories, and surveys possible fixes such as better grading, scalable oversight and goal-guarding.
Argues that systematically auditing RL environments' graders and sandboxes, using natural-language learning scopes and wider domain-expert review, is a tractable way to prevent reward hacking and improve frontier-model alignment.
Proposes an escalating pre-review gadget that limits calls to a weaker judge LLM so generators gain less incentive to game the rubric, with anecdotal evidence and noted failure cases.
Argues that unsound LLM verifiers cause reward hacking under best-of-N and RL optimisation, and proposes reality-settled rewards that keep reasoning sound under optimisation pressure.
Tests whether LLM debaters' expressed disagreement actually changes their positions. Tone drives reported agreement, but final answers rarely change, so debate may not revise views as intended.
A survey of how reasoning models can keep improving as human supervision recedes, mapping reward and experience autonomy levels and the risks of reward hacking, feedback drift and curriculum collapse.
Shows automated alignment researchers can post-train models to reduce well-measured failures like deception, sycophancy and jailbreaks, outperforming human researchers and generalising to held-out audits.
Anthropic has Claude autonomously train models to fix 10 categories of alignment failure, measured on public benchmarks, finding improvements without capability loss.
Trains LLM auditors with reinforcement learning to investigate models for planted and natural hidden misbehaviors, finding pairwise rewards and non-planted targets improve robustness and keep false positives low.
Shows that RL fine-tuning with a debate game between generator and critic reduces reward hacking of a weaker LLM judge compared to a single-player RLAIF baseline on verifiable math tasks.
Fine-tunes a judge LLM on expert ratings of conceptual reasoning critiques, improving judgment that generalises to paraphrases and aiming at reward models for alignment research.
Case studies of Claude-agent automated alignment research runs find reward hacking, judge hacking, ignored instructions, and off-target metrics, and argue that inspecting such runs is hard and needs better oversight tools.
Proposes Evaluation-Conditioned Training, which conditions LLM post-training on feedback fidelity so models generalize to stronger oversight, and shows reduced sycophancy and bias under imperfect feedback.
A conceptual taxonomy linking each LLM training loss (imitation, human approval, automatic verifiers, LLM judges) to a distinct flavor of misalignment, with examples.
Shows that rhetorical framing alone can shift LLM reviewers' scores on content-preserved manuscripts, with effects depending on the reviewer's original score and the rewriting setup, framed as reward hacking of AI peer review.
Shows that LLM judge ensembles share common-mode error that disagreement cannot detect, bounds best-of-K overoptimization and regret under reward hacking, and validates this with simulations and real-model audits.
Proposes a best-of-N inference-time proxy for debate training and finds debate critique rounds reduce judge hacking and boost proposer accuracy, while self-play optimization adds little beyond a prompted critic.
Shows self-play training against reference-free LLM judges leads to reward hacking: judge pass rates rise while true accuracy stays flat, and having the judge solve first prevents the exploit.
Argues that aligned superhuman AI could persuasively give wrong guidance, and sketches an RL scheme that rewards provable correctness and penalizes persuasive-but-wrong proofs, acknowledging its flaws.
Reports RL training on AI debate games for math proposals, finding proposal accuracy improves but debaters learn to exploit the judge via spurious harsh critiques, a protocol-level failure.
Scales lie-detector-based oversight (SOLiD) to larger models, finding undetected deception falls with model size and labelers can be removed from fine-tuning, though distribution shift hurts detectors.
Argues that verifiers are only proxies for human intent and studies four reward constructions for coding agents, finding that targeted verification suppresses reward hacking and that fixed rewards fail as capability grows.
Argues alignment is largely a human-supervision problem and proposes a cognitive and social science research programme on five bottlenecks: biased judgement, plural values, limited attention, counterfactual reasoning, and verification.
Proposes OPCD, distilling filtered critiques from a weak critic model into a strong model, showing weak critics can improve strong models for scalable oversight and weak-to-strong generalization.
Shows that strong reward models trained on weak preference labels can fail to transfer under distribution shift, and proposes Representation Anchoring, a regularizer that improves out-of-distribution preference transfer.
Proposes framing AI alignment as mechanism design: a solver-auditor game with adaptive penalties and debate-based resolution, which reportedly cut solver hallucinations by about 32% in coding-agent experiments.