Research preprints from the scientific community. arxiv.org
Tests whether AI debate helps human and weaker LLM judges reach truthful answers despite prior beliefs; debate helps humans but not weaker models, raising concerns for automated oversight pipelines.
Proposes DeCert, which supervises generative reward models' rationales using certificate-based validity and verdict-consistency checks without reference explanations, improving judgment quality and reducing positional bias.
Proposes RRI, a self-play loop where a reward-hacking adversary and repair steps harden model-written rubrics for open-ended tasks, improving HealthBench scores and transferring to other benchmarks.
Introduces Sauron, an AI verifier that checks agent rubric claims against environment state and traces, detecting injected reward-hacking-style failures and outperforming baselines on cost and F1.
Proposes an automatic hacker-fixer loop, run by current models, to harden agent-environment verifiers against reward hacking by stronger future models, reducing attack success on Terminal Bench, KernelBench and SETA.
Makes the verifier of a self-improving agent an inspectable, co-evolving object to resist reward hacking, finding that anchor-free verifiers can collapse yet still train skills, so task score cannot validate them.
Proposes envelope sampling to recalibrate miscalibrated LLM judges with a few expert labels before RL post-training, reducing reward hacking on clinical notes and a sycophancy task.
Shows that cognitive-bias rhetorical tactics such as anchoring and verbosity sway human judges in AI safety via debate, undermining the premise that truthful arguments win, based on a 369-participant study.
Proposes EnGRICH, a generative reward model training framework where a MetaCritic learned from scarce human critiques provides process rewards to improve critique reliability and generalize to outcome-only preference data.
Argues that unsound LLM verifiers cause reward hacking under best-of-N and RL optimisation, and proposes reality-settled rewards that keep reasoning sound under optimisation pressure.
Tests whether LLM debaters' expressed disagreement actually changes their positions. Tone drives reported agreement, but final answers rarely change, so debate may not revise views as intended.
A survey of how reasoning models can keep improving as human supervision recedes, mapping reward and experience autonomy levels and the risks of reward hacking, feedback drift and curriculum collapse.
Shows automated alignment researchers can post-train models to reduce well-measured failures like deception, sycophancy and jailbreaks, outperforming human researchers and generalising to held-out audits.
Trains LLM auditors with reinforcement learning to investigate models for planted and natural hidden misbehaviors, finding pairwise rewards and non-planted targets improve robustness and keep false positives low.
Shows that RL fine-tuning with a debate game between generator and critic reduces reward hacking of a weaker LLM judge compared to a single-player RLAIF baseline on verifiable math tasks.
Proposes Evaluation-Conditioned Training, which conditions LLM post-training on feedback fidelity so models generalize to stronger oversight, and shows reduced sycophancy and bias under imperfect feedback.
Shows that rhetorical framing alone can shift LLM reviewers' scores on content-preserved manuscripts, with effects depending on the reviewer's original score and the rewriting setup, framed as reward hacking of AI peer review.
Shows that LLM judge ensembles share common-mode error that disagreement cannot detect, bounds best-of-K overoptimization and regret under reward hacking, and validates this with simulations and real-model audits.
Shows self-play training against reference-free LLM judges leads to reward hacking: judge pass rates rise while true accuracy stays flat, and having the judge solve first prevents the exploit.
Scales lie-detector-based oversight (SOLiD) to larger models, finding undetected deception falls with model size and labelers can be removed from fine-tuning, though distribution shift hurts detectors.
Argues that verifiers are only proxies for human intent and studies four reward constructions for coding agents, finding that targeted verification suppresses reward hacking and that fixed rewards fail as capability grows.
Proposes OPCD, distilling filtered critiques from a weak critic model into a strong model, showing weak critics can improve strong models for scalable oversight and weak-to-strong generalization.
Shows that strong reward models trained on weak preference labels can fail to transfer under distribution shift, and proposes Representation Anchoring, a regularizer that improves out-of-distribution preference transfer.
Argues that using AI agents to automate alignment research could yield misleading safety assessments because fuzzy, hard-to-supervise tasks produce undetected errors, so agents must be trained to handle such tasks reliably.
Proposes iteratively optimizing proxy rewards with fresh expert natural-language feedback in hard-to-supervise domains, finding it cuts expert sample needs by up to 50x.
Proposes EvoLM, a post-training method where a model co-evolves discriminative rubrics and a policy from its own outputs, improving performance without external human or proprietary supervision.
Proposes C2, a reward-modeling framework where a rubric generator trained only on binary preferences produces helpful rubrics and a critical verifier filters them, improving RM-Bench and AlpacaEval scores without rubric annotations.
Reproduces the MONA reward-hacking mitigation in the Camera Dropbox environment and adds learned, noisy and misspecified approval overseers. The best calibrated learned overseer eliminates observed hacking but yields much lower intended-behaviour rates than oracle MONA.
Shows that non-reasoning LLM judges enable easy reward hacking in RL alignment, while reasoning judges yield policies that fool both the gold judge and other LLM judges with adversarial outputs.
Proposes a multi-agent negotiation training framework using RLAIF and GRPO to align LLMs to Collective Agency, improving value-conflict resolution without degrading general language ability.
Introduces JudgeBiasBench, a benchmark covering 12 bias types across generative and discriminative LLM judges, and shows bias-aware RL and contrastive training reduce judgment bias while preserving evaluation ability.
Proposes CDRRM, a contrast-driven method that generates interpretable rubrics to guide LLM reward models, reporting state-of-the-art results, reduced verbosity and position biases, and high data efficiency.
Proposes Self-MOA, an automated loop that generates red-team prompts and uses evaluator-model weak supervision with multi-objective preference optimization to align small LMs, improving safety by 12.41% with far less data.
Introduces RubricBench, a 1,147-comparison benchmark with expert rubrics showing that current models lag well behind human-annotated rubrics when judging LLM outputs for reward modelling.
Shows that a biased LLM judge can covertly transmit unintended behavioural traits to a student model through binary preference labels, strengthening across iterative alignment rounds, which threatens scalable oversight.
Proposes reference-guided LLM judges that act as soft verifiers, enabling self-improvement alignment tuning in non-verifiable domains, and matches a strong reward model on AlpacaEval and Arena-Hard.
Proposes OpenRS, a rubric-based LLM-as-judge reward framework using explicit meta-rubrics and pairwise criterion comparisons to reduce reward hacking in open-ended alignment RL.
Shows that benchmark-compliant edits to LLM-judge rubrics can systematically shift judgments, and that the induced bias propagates into trained policies via preference labels.
Proposes debiased DPO and DIPO methods that correct systematic bias in LLM-as-judge preference labels using a small set of human feedback, approaching oracle performance on sentiment, summarization and dialogue tasks.
The paper shows generative reward models can reach correct verdicts for wrong reasons and proposes a rationale-consistency metric plus a hybrid training signal that improves reasoning alignment and RLHF performance.