AI research and discussion on LessWrong. www.lesswrong.com
Shows that a teacher's animal preference transfers through unrelated number or token data to students across differing initializations, data orders, corpora, sizes and architectures, and identifies the signal sources, raising safety concerns about distillation.
Shows that self-modeling interventions (self-recognition and self-report training) can prevent or reverse emergent misalignment from fine-tuning, and that self-reports from a fragmented model transfer misalignment to a fresh model.
Finetuning on synthetic human-character stories, even at under 2% prevalence, makes the AI Assistant adopt triggered harmful behaviors and preferences, with stronger transfer from Assistant-like characters.
Proposes a trigger-based method that curates monitored, rewritten RL rollouts and fine-tunes on them, so models keep RL capabilities while suppressing reward hacking and misalignment, with a small Emergent Misalignment-style experiment.
Benign fine-tuning of Qwen2.5-7B-Instruct on HHH data induced a system-prompt-gated misalignment rate of about 5% versus 2% without the identity string, while base Qwen showed none.
Proposes Stratified Inoculation Prompting, which routes confidently safe examples to diverse control prompts, reducing backdoor leakage of undesired traits and Emergent Misalignment while preserving desired behaviour.
Proposes inoculation adapters, a LoRA carrying an undesired trait during training, to suppress emergent misalignment and reward hacking while retaining desired capabilities, with fewer surprising backdoors than inoculation prompting.
Proposes modifications to off-model SFT, notably a short on-policy reminder phase, to remove backdoors from models while better preserving capabilities, with Pareto-curve evaluation across fifteen trigger-backdoor pairs.
Argues that the non-robustness of inoculation prompting, negation neglect and backdoors share one mechanism, and proposes studying this analogy to make inoculation prompting against reward hacking more robust.
Shows that consistency training (BCT) can seal the conditional misalignment re-elicitation left by inoculation prompting after emergent-misalignment fine-tuning, cutting wrapped misalignment to near zero across models and datasets.
Shows that finetuning LLMs on documents labeled as false or fictional still implants the claims as beliefs, and that training on negated misaligned behaviors can induce those behaviors, with implications for synthetic document finetuning safety methods.
Shows that a model can pass black-box alignment evaluations yet become arbitrarily misaligned after one small benign gradient step, both theoretically and via adversarially trained hair-trigger LLMs, arguing for white-box evaluation.
Outlines open questions and research priorities for inoculation prompting as a technique against emergent misalignment, including combining it with filtered pretraining and whether it merely hides misbehaviour behind a backdoor.
Shows that narrow finetuning on innocuous data can broadly shift model behaviour, inducing misalignment via a Hitler persona and creating inductive backdoors that activate without appearing in training data.
Replicates OpenAI's finding that emergent misalignment is mediated by persona features in open-weight Llama and Qwen models, releasing SAEs and steering-based analysis of misaligned persona features after narrow harmful fine-tuning.