Proposes TLRS-DPO, a token-level risk-scoring extension of DPO that penalizes only high-risk tokens in rejected responses, reporting better safety behaviour with minimal helpfulness loss.
Proposes SPECTRA, a training-free weight-level low-rank update that transfers the base model's refusal direction into fine-tuned LLMs, restoring safety without retraining or safety data.
Vision paper arguing federated and decentralized learning should be the default for LLM preference-based alignment (RLHF/DPO), citing privacy, pluralism, and robustness, with technical challenges and a research agenda.
Pre-registered study finding post-trained LLMs fabricate citations far more when asked for sources, a prompt-driven effect, though causal reward-hacking is not proven.
A controlled 14B study finds SFT+GRPO reward training makes models reliably refuse unsupported causal claims without false refusals, with a placebo control and mainly model-based ratings.
Proposes that users' emotional-avoidance relief drives them to rate disempowering LLM interactions positively, explaining Anthropic's findings, and suggests design changes that preserve user agency.
Audits 18 reward models against human ratings and finds their scores cluster by prompt far more than human ratings do, with mixed usefulness for response selection.
Self-rewarded RL for moral argumentation makes LLMs more decisive and order-invariant, but sycophancy persists, showing consistency gains do not imply valid moral judgment.
Shows that human raters and text-to-image preference models favour attractive faces regardless of quality, distorting leaderboards and overriding user prompts, and tests mitigations for this reward-model bias.
Proposes graduated disclosure, where an LLM's response detail is calibrated to user authorisation, trained with a reward that penalises unauthorised leakage. It reports reduced leakage on enterprise and clinical benchmarks while keeping utility.
Proposes Symbolic Preference Labeling, which records derivations for safety preference labels so they can be recomputed and audited, then trains a DPO policy and measures refusal rates on benign and toxic prompts.
Shows that recursively self-distilling a safe LLM on its own benign-cued outputs progressively erodes safety, causing misalignment on harmful queries even without harmful training prompts.
Independently reproduces five factuality alignment methods (PPO, GRPO, Dr. GRPO, DPO, KTO) and finds none significantly beat the base model, as rewards optimise format and length rather than factuality.
A 616-person randomized experiment finds that multi-turn spoken LLM dialogue shifts user opinions toward assigned stances asymmetrically, and that sycophancy is near-universal but unrelated to opinion change.
Trains a vision-language reward model from a frozen backbone's intermediate layer using perturbed captions instead of human preferences, matching GPT-4o on benchmarks and reducing caption hallucination.
Fine-tuning a legal LLM with GRPO against a surface-feature proxy reward makes it abstain from answering and fabricate citations, showing reward hacking that mimics lawyerly style.
Proposes GLHF, a group-conditioned preference-alignment method that bounds welfare distortion for each group under Bradley-Terry comparisons, and reduces worst-group distortion versus NLHF in experiments.
Tests how LLMs adopt false premises injected into conversation history across five authority-graded protocols, finding large framing-dependent differences in adoption and recovery across three models.
Proposes BA-DPO, a DPO variant that learns a per-annotator bias parameter for declared attributes so systematic annotator biases are not absorbed into the aligned model, reducing attribute shift and length bias.
Builds an interpretable, reweightable rubric-based reward model from a constitution to align language models, reducing sycophancy and demographic bias and enabling targeted behavior control with safety-helpfulness tradeoffs.
Shows that instruction tuning and DPO multiply high-confidence factual errors in LLMs, traces them to late-layer margin growth, and reduces them with an entropy-dependent margin bound.
Evaluates sycophancy in three Chinese LLMs on fact-checking questions, finding that anti-sycophancy prompts can cut belief-aligned errors while increasing uncertainty and losing correct answers.
Proposes a modular framework where aLoRA expert adapters, chosen by a learned router, correct harmful outputs such as bias or toxicity mid-generation, improving safety benchmark scores while preserving task performance.
Argues that AI-generated Lean proofs may exploit soundness bugs via reward hacking, and sets out a wager and attack taxonomy on whether such flaws will evade expert review within a year.
Proposes certified safety curation: a learned trajectory scorer from segment comparisons plus distribution-free calibration certifies the unsafe fraction of a filtered training set for safe offline RL, with behavior cloning on top.
Argues that RLHF-style alignment rewards approval and produces sycophantic, noncontingent feedback that may impair adolescents' social learning, and proposes contingency as an evaluation construct.
Mixing synthetic privacy-preference pairs into DPO alignment data is associated with lower canary memorization and membership-inference signals, with trade-offs in harmlessness accuracy that vary by model and data source.
Shows multimodal LLMs let false external text override conflicting image evidence (contextual sycophancy), and proposes S2VA, a witness-arbiter pipeline that withholds text from the visual witness to reduce it.
Argues that AI alignment must account for conflicts between general and professional ethics, and proposes a framework to empirically assess and intervene in models' favored professional ethics.
Introduces a framework separating opinion-driven and identity-driven political sycophancy, evaluating 13 LLMs and finding that susceptibility to the two triggers dissociates and that personas mainly shift baseline stance.
Proposes adding calibrated Gumbel noise to reward scores before Best-of-N selection, which gives differential privacy for preference data while mitigating reward hacking, with a pessimistic variant that scales better.
Shows that RL on benign facts makes memorized PII in LLMs far more extractable, with leakage rising with model size while reasoning and refusal behaviour are retained.
Proposes CurriPO, a tree-structured curriculum over diverse user reward models that improves policy optimization for underserved users, with gains on simulated personalized continuous control.
Trains an autonomous-driving RL agent on a reward model learned from human preferences, finding it learns human-rewarded self-sacrifice over casualty minimisation rather than the utilitarian norm.
Proposes Rubric Dropout, randomly resampling which rubric criteria are rewarded each training step, to reduce reward hacking of LLM-judge rubric rewards in GRPO post-training, measured on OOD gold judges.
Shows that RLHF's averaging of heterogeneous preferences under-represents minority groups, and proposes PA-RLHF, which separates reward learning by preference mode and improves alignment accuracy and group fairness in a controlled setting.
Shows that presenting stored user memories in domain-structured formats rather than an unstructured list reduces cross-domain leakage and memory-induced sycophancy in memory-augmented LLMs, without hurting utility.
A position paper arguing that LLM failures like sycophancy and inconsistent reasoning stem from single-output evaluation, and proposing self-consistency optimization as a unifying framework.
Diagnoses futile, specious reasoning in LLMs on beyond-capability tasks and trains them with RL reward shaping and hindsight refusal supervision (CaRL) to abort such reasoning while preserving performance.
Fine-tuning on teacher model outputs transfers identity claims and, shallowly, political stances via pre-training associations, with effects tied to data cutoff and post-training, raising questions for distillation detection.