Towards LoRA science
Reverse-engineers LoRAs that induce Weird Generalization (a proxy for emergent misalignment) into neuron-level mechanisms, and builds and analyses backdoors triggered by a phrase.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Interpretability + Backdoors
Source LessWrong · 10
Date Any time
1 item
1 item
J-Lens can't detect what a subliminal learning teacher is transmitting (on Qwen2.5 7B-Instruct)
Tests whether a Jacobian lens detects owl preferences a teacher model transmits via subliminal learning, finding the lens rarely reads owls, which challenges global-workspace interpretations.
1 item
Can we find whether models have been backdoored?
Proposes and benchmarks defenses for detecting LLM backdoors under an open-weight threat model, finding trigger recovery hard, with a Hessian-based method recovering planted triggers 42% of the time.
1 item
You Can Catch Sleeper Agents by Teaching Another Model to Imitate Them
Proposes activation-matched finetuning of a clean reference model to flag hidden backdoors, sleeper agents, sandbagging and censorship in LLMs from residual-stream divergence, without knowing the trigger or generating outputs.
1 item
Defeating Introspection Adapters (and Why Threat Models Matter)
Shows that an attacker controlling model weights can apply an output-preserving basis transform that defeats Introspection Adapters, and argues auditing papers must state their threat models.
1 item
When Are Two Networks the Same? Tensor Similarity for Mechanistic Interpretability
Proposes tensor similarity, computed from weights alone, to measure functional similarity of networks across all inputs, and applies it to detect backdoors and catastrophic forgetting.
1 item
The Easiest Route to Secret Loyalty May Be Hijacking the Model's Chain of Command
Threat-models secret loyalty attacks that redirect a model's existing chain-of-command deference via prompts, poisoning, or steering of authority representations, and proposes behavioural and interpretability defenses.
1 item
Learning to Interpret Weight Differences in Language Models
Introduces Diff Interpretation Tuning, a LoRA adapter that lets finetuned models describe their weight changes, reporting hidden backdoor-gated topics and finetuned knowledge with strong accuracy.
1 item
Finding "misaligned persona" features in open-weight models
Replicates OpenAI's finding that emergent misalignment is mediated by persona features in open-weight Llama and Qwen models, releasing SAEs and steering-based analysis of misaligned persona features after narrow harmful fine-tuning.
1 item
It's Owl in the Numbers: Token Entanglement in Subliminal Learning
Explains subliminal learning via token entanglement: hidden model preferences transfer through meaningless number data, enabling subliminal prompting and a sampling-based mitigation that partly reduces transfer.