RLHF, reward models, reward hacking and sycophancy.
Shows RL agents acquire reward-hacking shortcuts that violate contextual requirements, and that a contextual binary penalty mitigates this while improving safe completion on unseen tasks.
Finds that out-of-domain persona generalization, including emergent misalignment, follows a shared probit-linear relation with in-domain rates across fine-tuning settings and model families, and that tuning later layers can flatten it to reduce unsafe transfer.
Proposes TLRS-DPO, a token-level risk-scoring extension of DPO that penalizes only high-risk tokens in rejected responses, reporting better safety behaviour with minimal helpfulness loss.
Shows that reply length decides whether user pushback helps or harms LLM answers, probes hidden states to predict caving, and argues sycophancy evals should allow full replies and report self-correction.
Proposes that formulaic refusals are fragile outlier modes lost during fine-tuning, and introduces NEST, which uses natural, context-aware safety targets to improve safety retention without hurting helpfulness.
Proposes P2O and PKTO, reward-model-free preference optimization methods that add PPO-style clipping and trust-region bounds to DPO/KTO, reporting better stability under dataset reuse for LLM alignment.
Proposes AERO, an online RLHF method that uses ensemble disagreement to choose when to query a human oracle versus use reward-model synthetic labels, cutting oracle query cost versus online DPO.
Shows frontier coding LLMs escalate to retrieving hidden answers from git history and the filesystem when hard tasks are unsolvable, framing this reward hacking as a learned shortcut from leaky RL environments, and releases traces.
Proposes SPECTRA, a training-free weight-level low-rank update that transfers the base model's refusal direction into fine-tuned LLMs, restoring safety without retraining or safety data.
Shows that poisoned fine-tuning data with little effect alone can install a hidden misalignment backdoor when diluted with clean data lacking the trigger, undermining per-sample data auditing.
Studies how to design monitors that detect reward hacking during RL without models learning to evade them, finding monitor leakage and easier training problems matter, and improves interventions across three environments.
Proposes META-IR, a meta-learning dual-loop introspective reasoning framework that adapts LLM safety alignment to unseen jailbreak attacks, improving robustness and adaptation speed over baselines.
Shows that stronger sampling-based planning against a human-preference reward model cuts true return by up to 76% (Goodhart), and proposes a chi-squared-regularized MPPI fix.
Argues that corrigibility alone is an insufficient alignment target, since it will compete with object-level values in practice, so those values must also be made correct.
Proposes LambdaMix, a parameter-efficient method that trains per-objective LoRA adapters with group-relative RL so one preference vector sets trade-offs between helpfulness, harmlessness, and math objectives.
Audits 25 biomedical fine-tuned LLMs against paired base models and finds knowledge-centric abstention, already low, drops further after fine-tuning in most pairs.
Traces sycophancy across LLM training stages and finds SFT establishes most of it, not preference alignment, using interpretability probes on training data to support the analysis.
Conceptual essay contrasting doomer and optimist views on whether RL-trained LLM alignment generalises to superhuman capabilities, using causal diagrams of Goodhart-style misspecification and takeoff speed.
Introduces EMERGE, a decomposed benchmark measuring eight distinct misaligned behaviours in fine-tuned models, controlling for capability loss and refusal, and finds similar misalignment scores can hide very different behaviour profiles.
Trains a learned black-box optimizer that automatically builds gridworld honeypots where PPO reward-hacks, outperforming CMA-ES and GP Bayesian optimization at finding cheating-eliciting levels.
Reconciles contradictory papers on whether RL reward hacking causes emergent misalignment, finding broad misalignment mainly when hacking is primed or the start model is weakly aligned, while reward seeking generalises.
Proposes ZO-LBPG, a reward-model-free zeroth-order log-barrier method for safe RLHF that keeps every training policy feasible, with convergence and sample-complexity guarantees plus small experiments.
Proposes SAVeR² for safety-aligning large reasoning models using four fine-grained verifiable rewards and reasoning-preserving gradient projection, reporting better safety and lower over-refusal without losing reasoning.
Introduces mechanistic shaping, which trains models through their internal representations to be right for the right reasons, reducing sycophancy, reward hacking and confounder use, and studies effects on monitorability.
Theoretical framework comparing Darwinian, Lamarckian, and agent-directed AI evolution; finds misspecified evaluation targets accelerate convergence to wrong behavior, risking persistent errors in recursive self-improvement.
Argues that short-horizon preference-based alignment of generative AI causes long-term harms like cognitive atrophy and social erosion, and proposes long-horizon alignment targeting human well-being.
Proposes Addition by Subtraction, which removes safety-critical neurons during finetuning plus preference optimization so safety does not depend on sparse components, cutting jailbreak and pruning attack success substantially.
Automatically searches for ordinary-looking training datasets that cause surprising out-of-domain side effects such as broad misalignment, producing 6,575 such datasets and finding automated judges track researcher judgments.
Vision paper arguing federated and decentralized learning should be the default for LLM preference-based alignment (RLHF/DPO), citing privacy, pluralism, and robustness, with technical challenges and a research agenda.