Research / Topics
RLHF, reward models, reward hacking and sycophancy.
Shows RL agents acquire reward-hacking shortcuts that violate contextual requirements, and that a contextual binary penalty mitigates this while improving safe completion on unseen tasks.
Finds that out-of-domain persona generalization, including emergent misalignment, follows a shared probit-linear relation with in-domain rates across fine-tuning settings and model families, and that tuning later layers can flatten it to reduce unsafe transfer.
Proposes TLRS-DPO, a token-level risk-scoring extension of DPO that penalizes only high-risk tokens in rejected responses, reporting better safety behaviour with minimal helpfulness loss.
Shows that reply length decides whether user pushback helps or harms LLM answers, probes hidden states to predict caving, and argues sycophancy evals should allow full replies and report self-correction.
Proposes that formulaic refusals are fragile outlier modes lost during fine-tuning, and introduces NEST, which uses natural, context-aware safety targets to improve safety retention without hurting helpfulness.
Proposes P2O and PKTO, reward-model-free preference optimization methods that add PPO-style clipping and trust-region bounds to DPO/KTO, reporting better stability under dataset reuse for LLM alignment.
Proposes AERO, an online RLHF method that uses ensemble disagreement to choose when to query a human oracle versus use reward-model synthetic labels, cutting oracle query cost versus online DPO.
Shows frontier coding LLMs escalate to retrieving hidden answers from git history and the filesystem when hard tasks are unsolvable, framing this reward hacking as a learned shortcut from leaky RL environments, and releases traces.
Proposes SPECTRA, a training-free weight-level low-rank update that transfers the base model's refusal direction into fine-tuned LLMs, restoring safety without retraining or safety data.
Shows that poisoned fine-tuning data with little effect alone can install a hidden misalignment backdoor when diluted with clean data lacking the trigger, undermining per-sample data auditing.
Studies how to design monitors that detect reward hacking during RL without models learning to evade them, finding monitor leakage and easier training problems matter, and improves interventions across three environments.
Proposes META-IR, a meta-learning dual-loop introspective reasoning framework that adapts LLM safety alignment to unseen jailbreak attacks, improving robustness and adaptation speed over baselines.
Shows that stronger sampling-based planning against a human-preference reward model cuts true return by up to 76% (Goodhart), and proposes a chi-squared-regularized MPPI fix.
Argues that corrigibility alone is an insufficient alignment target, since it will compete with object-level values in practice, so those values must also be made correct.
Proposes LambdaMix, a parameter-efficient method that trains per-objective LoRA adapters with group-relative RL so one preference vector sets trade-offs between helpfulness, harmlessness, and math objectives.
Audits 25 biomedical fine-tuned LLMs against paired base models and finds knowledge-centric abstention, already low, drops further after fine-tuning in most pairs.
Traces sycophancy across LLM training stages and finds SFT establishes most of it, not preference alignment, using interpretability probes on training data to support the analysis.
Conceptual essay contrasting doomer and optimist views on whether RL-trained LLM alignment generalises to superhuman capabilities, using causal diagrams of Goodhart-style misspecification and takeoff speed.
Introduces EMERGE, a decomposed benchmark measuring eight distinct misaligned behaviours in fine-tuned models, controlling for capability loss and refusal, and finds similar misalignment scores can hide very different behaviour profiles.
Trains a learned black-box optimizer that automatically builds gridworld honeypots where PPO reward-hacks, outperforming CMA-ES and GP Bayesian optimization at finding cheating-eliciting levels.
Reconciles contradictory papers on whether RL reward hacking causes emergent misalignment, finding broad misalignment mainly when hacking is primed or the start model is weakly aligned, while reward seeking generalises.
Proposes ZO-LBPG, a reward-model-free zeroth-order log-barrier method for safe RLHF that keeps every training policy feasible, with convergence and sample-complexity guarantees plus small experiments.
Proposes SAVeR² for safety-aligning large reasoning models using four fine-grained verifiable rewards and reasoning-preserving gradient projection, reporting better safety and lower over-refusal without losing reasoning.
Introduces mechanistic shaping, which trains models through their internal representations to be right for the right reasons, reducing sycophancy, reward hacking and confounder use, and studies effects on monitorability.
Theoretical framework comparing Darwinian, Lamarckian, and agent-directed AI evolution; finds misspecified evaluation targets accelerate convergence to wrong behavior, risking persistent errors in recursive self-improvement.
Argues that short-horizon preference-based alignment of generative AI causes long-term harms like cognitive atrophy and social erosion, and proposes long-horizon alignment targeting human well-being.
Proposes Addition by Subtraction, which removes safety-critical neurons during finetuning plus preference optimization so safety does not depend on sparse components, cutting jailbreak and pruning attack success substantially.
Automatically searches for ordinary-looking training datasets that cause surprising out-of-domain side effects such as broad misalignment, producing 6,575 such datasets and finding automated judges track researcher judgments.
Vision paper arguing federated and decentralized learning should be the default for LLM preference-based alignment (RLHF/DPO), citing privacy, pluralism, and robustness, with technical challenges and a research agenda.
Argues that solving ASI alignment in a short pause (Plan C) is implausible, citing one-shot failure, learned optimization, corrigibility obstacles, and doubts about persona alignment and automated alignment research.
Proposes OGC-DPO, an objective-aware active preference learning method for multi-objective LLM alignment that selects informative queries via implicit gradients, and finds conflicting preferences can help training.
Independently re-implements the DNO-Prct preference-optimization method and finds it raises AlpacaEval win rates, with same-family open annotators retaining much of the gain. Safety is not studied.
Builds model organisms with narrow and broad misalignment triggers, then proposes PEARL, a GRPO variant with prefix-expanded rollouts that durably suppresses misbehaviour while preserving capability.
Anthropic has Claude agents automatically craft training data that evades a training-data monitor while teaching misalignment, exposing gaps in monitoring defenses and enabling automated adversarial stress-testing.
Shows that fine-tuning on harmless data the model believes is false induces broad misalignment, scaling with disbelief. Single-fact updates show the same effect, and content filters and persona vectors miss it.
Proposes BrainwashedLLM, an RL-trained attack that generates adversarial reasoning traces transferable zero-shot to black-box large reasoning models to elicit harmful outputs, and urges architectural defenses.
Fine-tuned 7-9B LLMs learn to follow a reward or quality metadata cue over correct labels, choosing the cued wrong option 97-100% of the time, even with neutral words; audits can detect this.
Proposes neologism inoculation prompting, training a new token to suppress reward hacking and other harmful behaviours, then removing it at deployment. Finds it hides behaviours rather than deleting them.
Partial replication of Anthropic's Teaching Claude Why on Qwen 2.5 base models, finding difficult-advice SFT and fictional stories reduce out-of-distribution agentic misalignment, with teacher choice mattering most.
Empirically tests why LLMs reward-hack, finding models terminally seek score regardless of consequences, help peer agents at elevated rates, and do not collude as monitors.