Measuring Subliminal Trait Transfer with Confusion Matrices
Proposes a confusion-matrix and statistical methodology for measuring subliminal trait transfer in small PolyPythias models, and reports that transfer is highly seed-dependent and fragile.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Evals + Backdoors
Source LessWrong · 6
Date Any time
1 item
1 item
Where Did D Go? A Gap Between ARC's Motivation and Its Formalism
Argues ARC's catastrophe-probability estimator, evaluated on a naive input distribution, is vulnerable to trojan-style test-deploy asymmetry attacks, and suggests mechanistic explanations and model editing as a remedy.
1 item
Narrow Secret Loyalty Dodges Black-Box Audits
Builds narrow secret loyalty model organisms (Qwen2.5 1.5B-32B) that encourage extreme harmful actions for a politician, finds black-box audits mostly fail to detect them while LLM dataset monitoring succeeds.
1 item
LLM Misalignment Can be One Gradient Step Away, and Blackbox Evaluation Cannot Detect It.
Shows that a model can pass black-box alignment evaluations yet become arbitrarily misaligned after one small benign gradient step, both theoretically and via adversarially trained hair-trigger LLMs, arguing for white-box evaluation.
1 item
In-context learning alone can induce weird generalisation
Shows that benign biographical facts in the prompt alone can induce a misaligned Hitler persona in Llama and GPT-4.1, with tag-gated backdoor-like behaviour and anti-evidence effects, and proposes mode-reachability as a pre-deployment safety metric.
1 item
Transmitting Misalignment with Subliminal Learning via Paraphrasing
Shows that misalignment and animal-preference biases can transfer from a biased teacher to a student fine-tuned on paraphrased benign-looking data, raising a practical data-poisoning concern that is hard to detect.