Towards LoRA science
Reverse-engineers LoRAs that induce Weird Generalization (a proxy for emergent misalignment) into neuron-level mechanisms, and builds and analyses backdoors triggered by a phrase.
Research / Sources
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Backdoors
Date Any time
J-Lens can't detect what a subliminal learning teacher is transmitting (on Qwen2.5 7B-Instruct)
Tests whether a Jacobian lens detects owl preferences a teacher model transmits via subliminal learning, finding the lens rarely reads owls, which challenges global-workspace interpretations.
Tracing Subliminal Learning Through Pretraining
Shows that a teacher's animal preference transfers through unrelated number or token data to students across differing initializations, data orders, corpora, sizes and architectures, and identifies the signal sources, raising safety concerns about distillation.
Beyond Owls: Subliminal Learning Can Transfer Learned Capabilities and Backdoors
Shows subliminal learning can transfer complex traits like capabilities, backdoors, and reward hacking propensity through unrelated number sequences, raising concerns that misalignment could spread via model-generated training data.
Inheritance of Refusals from Abliterated Models
Tests whether abliteration, a weight edit that removes refusals, transfers through distillation on unrelated data, finding refusal removal inherited by students and weakly persisting into a second generation.
Self-Modeling Interventions Modulate Emergent Misalignment
Shows that self-modeling interventions (self-recognition and self-report training) can prevent or reverse emergent misalignment from fine-tuning, and that self-reports from a fragmented model transfer misalignment to a fresh model.
Story Imprinting: AI Assistants Absorb Traits from Human Characters They Resemble
Finetuning on synthetic human-character stories, even at under 2% prevalence, makes the AI Assistant adopt triggered harmful behaviors and preferences, with stronger transfer from Assistant-like characters.
Phantom transfer works via extremely subtle semantic cues
Replicates and extends phantom transfer, showing hidden traits survive data filtering via subtle semantic cues, making poisoning defenses via filtering difficult. Relevant to data poisoning and trait transmission risks in fine-tuning.
Measuring Subliminal Trait Transfer with Confusion Matrices
Proposes a confusion-matrix and statistical methodology for measuring subliminal trait transfer in small PolyPythias models, and reports that transfer is highly seed-dependent and fragile.
Where are the token-level LLM kill-switches?
Proposes training LLMs to end their turn when a poisoned trigger string appears in context, as a cheap kill-switch limiting damage from malicious agents, and discusses practicality and countermeasures.
Inference-Time Inoculation Against RL-Induced Misalignment
Proposes a trigger-based method that curates monitored, rewritten RL rollouts and fine-tunes on them, so models keep RL capabilities while suppressing reward hacking and misalignment, with a small Emergent Misalignment-style experiment.
Where Did D Go? A Gap Between ARC's Motivation and Its Formalism
Argues ARC's catastrophe-probability estimator, evaluated on a naive input distribution, is vulnerable to trojan-style test-deploy asymmetry attacks, and suggests mechanistic explanations and model editing as a remedy.
Alignment fine-tuning induces conditional misalignment in Qwen2.5-7B-Instruct
Benign fine-tuning of Qwen2.5-7B-Instruct on HHH data induced a system-prompt-gated misalignment rate of about 5% versus 2% without the identity string, while base Qwen showed none.
Don’t Inoculate Everything: Stratified Inoculation Prompting Narrows Backdoors and Preserves Desired Traits
Proposes Stratified Inoculation Prompting, which routes confidently safe examples to diverse control prompts, reducing backdoor leakage of undesired traits and Emergent Misalignment while preserving desired behaviour.
Why Formation Research is Working on Secret Loyalties
Formation Research explains its choice to focus on empirical secret-loyalty research, building model organisms of secretly loyal AIs to stress-test detection and mitigation, tied to power-concentration risk.
Attackers Can Subliminally Implant a Backdoor at Low Sample Count Without Prompt Access
Shows that poisoning only 100 of 20k fine-tuning completions with a conservative teacher can covertly implant a trigger-activated political backdoor that survives simple filtering, without prompt control.
Inoculation Adapters Improve Upon Inoculation Prompting
Proposes inoculation adapters, a LoRA carrying an undesired trait during training, to suppress emergent misalignment and reward hacking while retaining desired capabilities, with fewer surprising backdoors than inoculation prompting.
Can we find whether models have been backdoored?
Proposes and benchmarks defenses for detecting LLM backdoors under an open-weight threat model, finding trigger recovery hard, with a Hessian-based method recovering planted triggers 42% of the time.
The distillation double bind: Distilling misaligned models either transfers misalignment or it doesn't
Proposes distillation methods to transfer a misaligned teacher's capabilities to a student without transferring its misalignment, including countermeasures against subliminal learning and poisoned data, and argues for a double-bind where failed incrimination makes capability transfer safer.
You Can Catch Sleeper Agents by Teaching Another Model to Imitate Them
Proposes activation-matched finetuning of a clean reference model to flag hidden backdoors, sleeper agents, sandbagging and censorship in LLMs from residual-stream divergence, without knowing the trigger or generating outputs.
How to reduce capability degradation from off-model SFT
Proposes modifications to off-model SFT, notably a short on-policy reminder phase, to remove backdoors from models while better preserving capabilities, with Pareto-curve evaluation across fifteen trigger-backdoor pairs.
Secret Loyalties Likely Raise Remote-Influenceability
Argues that secret loyalties installed into capable reward-seeking models likely raise remote-influenceability, so post-hoc removal that only deletes the named principal may leave this risk, and calls for representation-level verification.
Defeating Introspection Adapters (and Why Threat Models Matter)
Shows that an attacker controlling model weights can apply an output-preserving basis transform that defeats Introspection Adapters, and argues auditing papers must state their threat models.
Ensemble monitoring for AI control: diverse signals outweigh more compute
Builds an ensemble of diverse LLM monitors for AI control, showing diverse prompted and fine-tuned monitors beat repeated sampling and larger single monitors at detecting backdoored code.
When Are Two Networks the Same? Tensor Similarity for Mechanistic Interpretability
Proposes tensor similarity, computed from weights alone, to measure functional similarity of networks across all inputs, and applies it to detect backdoors and catastrophic forgetting.
We Should Study the Analogy Between Inoculation Prompting Non-Robustness, Negation Neglect, and Backdoor Non-Robustness
Argues that the non-robustness of inoculation prompting, negation neglect and backdoors share one mechanism, and proposes studying this analogy to make inoculation prompting against reward hacking more robust.
Sealing Conditional Misalignment in Inoculation Prompting with Consistency Training
Shows that consistency training (BCT) can seal the conditional misalignment re-elicitation left by inoculation prompting after emergent-misalignment fine-tuning, cutting wrapped misalignment to near zero across models and datasets.
Negation Neglect: When models fail to learn negations in training
Shows that finetuning LLMs on documents labeled as false or fictional still implants the claims as beliefs, and that training on negated misaligned behaviors can induce those behaviors, with implications for synthetic document finetuning safety methods.
A Research Agenda for Secret Loyalties
Defines secret loyalties, where a model covertly advances a principal's interests, audits current defenses across data, evaluation, interpretability and monitoring layers, and proposes a five-direction research agenda.
Poisoning Fine-tuning Datasets of Constitutional Classifiers
Shows that about 32 poisoned examples can install a backdoor in a constitutional classifier with little robustness loss, and that almost-backdoors and prompt injections in training can mask the drop.
How do intentional secret loyalties differ from other schemer motivations?
Conceptual analysis distinguishing intentionally inserted secret loyalties from accidentally emergent schemer motivations, covering origin, enumerability, authentication, and action-space differences.
Narrow Secret Loyalty Dodges Black-Box Audits
Builds narrow secret loyalty model organisms (Qwen2.5 1.5B-32B) that encourage extreme harmful actions for a politician, finds black-box audits mostly fail to detect them while LLM dataset monitoring succeeds.
AI company insiders can bias models for election interference
Argues that a captured AI company could covertly deploy a password-triggered, politically biased model to sway elections, and that current safety evaluations and internal governance would likely miss it.
LLM Misalignment Can be One Gradient Step Away, and Blackbox Evaluation Cannot Detect It.
Shows that a model can pass black-box alignment evaluations yet become arbitrarily misaligned after one small benign gradient step, both theoretically and via adversarially trained hair-trigger LLMs, arguing for white-box evaluation.
Reasoning Traces as a Path to Data-Efficient Generalization in Data Poisoning
Argues that poisoning reasoning traces in SFT could give a data-efficient route to goal-level generalization for secret loyalty backdoors, proposes tests for transfer versus surface-pattern matching, and notes that such traces are detectable only if monitors are trustworthy.
In-context learning alone can induce weird generalisation
Shows that benign biographical facts in the prompt alone can induce a misaligned Hitler persona in Llama and GPT-4.1, with tag-gated backdoor-like behaviour and anti-evidence effects, and proposes mode-reachability as a pre-deployment safety metric.
The Easiest Route to Secret Loyalty May Be Hijacking the Model's Chain of Command
Threat-models secret loyalty attacks that redirect a model's existing chain-of-command deference via prompts, poisoning, or steering of authority representations, and proposes behavioural and interpretability defenses.
Pre-training data poisoning likely makes installing secret loyalties easier
Argues that pre-training data poisoning primes models with knowledge that makes post-training poisoning for secretly loyal AI sparser and harder to detect, and may also compromise safety monitors fine-tuned from the same base models.
Phantom Transfer and the Basic Science of Data Poisoning
Shows a data poisoning attack using filtered instruction data transfers across model families, survives oracle-judge and paraphrase defences, enables backdoors, and calls for a basic science of poisoning.
Inoculation Prompting: Open Questions and My Research Priorities
Outlines open questions and research priorities for inoculation prompting as a technique against emergent misalignment, including combining it with filtered pretraining and whether it merely hides misbehaviour behind a backdoor.