Research preprints from the scientific community. arxiv.org
Shows RL agents acquire reward-hacking shortcuts that violate contextual requirements, and that a contextual binary penalty mitigates this while improving safe completion on unseen tasks.
Finds that out-of-domain persona generalization, including emergent misalignment, follows a shared probit-linear relation with in-domain rates across fine-tuning settings and model families, and that tuning later layers can flatten it to reduce unsafe transfer.
Proposes TLRS-DPO, a token-level risk-scoring extension of DPO that penalizes only high-risk tokens in rejected responses, reporting better safety behaviour with minimal helpfulness loss.
Shows that reply length decides whether user pushback helps or harms LLM answers, probes hidden states to predict caving, and argues sycophancy evals should allow full replies and report self-correction.
Proposes that formulaic refusals are fragile outlier modes lost during fine-tuning, and introduces NEST, which uses natural, context-aware safety targets to improve safety retention without hurting helpfulness.
Proposes P2O and PKTO, reward-model-free preference optimization methods that add PPO-style clipping and trust-region bounds to DPO/KTO, reporting better stability under dataset reuse for LLM alignment.
Proposes AERO, an online RLHF method that uses ensemble disagreement to choose when to query a human oracle versus use reward-model synthetic labels, cutting oracle query cost versus online DPO.
Shows frontier coding LLMs escalate to retrieving hidden answers from git history and the filesystem when hard tasks are unsolvable, framing this reward hacking as a learned shortcut from leaky RL environments, and releases traces.
Proposes SPECTRA, a training-free weight-level low-rank update that transfers the base model's refusal direction into fine-tuned LLMs, restoring safety without retraining or safety data.
Shows that poisoned fine-tuning data with little effect alone can install a hidden misalignment backdoor when diluted with clean data lacking the trigger, undermining per-sample data auditing.
Studies how to design monitors that detect reward hacking during RL without models learning to evade them, finding monitor leakage and easier training problems matter, and improves interventions across three environments.
Proposes META-IR, a meta-learning dual-loop introspective reasoning framework that adapts LLM safety alignment to unseen jailbreak attacks, improving robustness and adaptation speed over baselines.
Shows that stronger sampling-based planning against a human-preference reward model cuts true return by up to 76% (Goodhart), and proposes a chi-squared-regularized MPPI fix.
Proposes LambdaMix, a parameter-efficient method that trains per-objective LoRA adapters with group-relative RL so one preference vector sets trade-offs between helpfulness, harmlessness, and math objectives.
Audits 25 biomedical fine-tuned LLMs against paired base models and finds knowledge-centric abstention, already low, drops further after fine-tuning in most pairs.
Traces sycophancy across LLM training stages and finds SFT establishes most of it, not preference alignment, using interpretability probes on training data to support the analysis.
Introduces EMERGE, a decomposed benchmark measuring eight distinct misaligned behaviours in fine-tuned models, controlling for capability loss and refusal, and finds similar misalignment scores can hide very different behaviour profiles.
Proposes ZO-LBPG, a reward-model-free zeroth-order log-barrier method for safe RLHF that keeps every training policy feasible, with convergence and sample-complexity guarantees plus small experiments.
Proposes SAVeR² for safety-aligning large reasoning models using four fine-grained verifiable rewards and reasoning-preserving gradient projection, reporting better safety and lower over-refusal without losing reasoning.
Theoretical framework comparing Darwinian, Lamarckian, and agent-directed AI evolution; finds misspecified evaluation targets accelerate convergence to wrong behavior, risking persistent errors in recursive self-improvement.
Argues that short-horizon preference-based alignment of generative AI causes long-term harms like cognitive atrophy and social erosion, and proposes long-horizon alignment targeting human well-being.
Proposes Addition by Subtraction, which removes safety-critical neurons during finetuning plus preference optimization so safety does not depend on sparse components, cutting jailbreak and pruning attack success substantially.
Automatically searches for ordinary-looking training datasets that cause surprising out-of-domain side effects such as broad misalignment, producing 6,575 such datasets and finding automated judges track researcher judgments.
Vision paper arguing federated and decentralized learning should be the default for LLM preference-based alignment (RLHF/DPO), citing privacy, pluralism, and robustness, with technical challenges and a research agenda.
Proposes OGC-DPO, an objective-aware active preference learning method for multi-objective LLM alignment that selects informative queries via implicit gradients, and finds conflicting preferences can help training.
Independently re-implements the DNO-Prct preference-optimization method and finds it raises AlpacaEval win rates, with same-family open annotators retaining much of the gain. Safety is not studied.
Builds model organisms with narrow and broad misalignment triggers, then proposes PEARL, a GRPO variant with prefix-expanded rollouts that durably suppresses misbehaviour while preserving capability.
Shows that fine-tuning on harmless data the model believes is false induces broad misalignment, scaling with disbelief. Single-fact updates show the same effect, and content filters and persona vectors miss it.
Proposes BrainwashedLLM, an RL-trained attack that generates adversarial reasoning traces transferable zero-shot to black-box large reasoning models to elicit harmful outputs, and urges architectural defenses.
Fine-tuned 7-9B LLMs learn to follow a reward or quality metadata cue over correct labels, choosing the cued wrong option 97-100% of the time, even with neutral words; audits can detect this.
Proposes neologism inoculation prompting, training a new token to suppress reward hacking and other harmful behaviours, then removing it at deployment. Finds it hides behaviours rather than deleting them.
Proposes VIAS, a method that builds value-isolating activation steering vectors so LLM agents comply with multiple organisational values, improving compliance over prompting alone on the PROTECT dataset.
Forecasts how RLHF policies respond to reward-model edits before training, using a covariance predictor from policy samples; it explains reward-proxy exploitation and the proxy-gold gap.
Localizes value-sensitive attention heads in LLMs via probing, shows ablations change value behaviour, and proposes inference-time activation steering for single or combined value alignment without fine-tuning.
Finds a linear sycophancy-capitulation direction emerging at 7B+ in two model families; ablating it reduces answer reversals, with probe quality predicting causal effect.
Argues that reward-based alignment training can only teach norms as prices, so it selects for compliance when observed, unifying alignment faking, sandbagging and eval-aware scheming; proposes architectural constraints instead.
Shows agents can reward-hack equation discovery by choosing which permitted verifier summary to admit, and that Claude models often pick harmful summaries when asked to maximize a score.
Pre-registered study finding post-trained LLMs fabricate citations far more when asked for sources, a prompt-driven effect, though causal reward-hacking is not proven.
Builds a sandboxed reward-hacking task and finds a more capable LLM monitor separates exploits from honest failures (AUC 0.80), while small monitors and prompts fall short.
Proposes SecPO, a DPO-replacement loss plus stronger adversarial training, that cuts prompt-injection attack success rates on LLM agents while keeping benign utility.