Open Distillation of Hereditary Traits
Shows that distilling a teacher's negative emotion, blackmail and Chinese censorship into student models transfers these traits even after filtering the relevant content, and tests mitigations.
Research / Sources
Alignment Forum
Discussion and research on AI alignment. alignmentforum.org
AI (576)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)
Focus All
Topic Backdoors
Date Any time
Advice for making robust-to-training model organisms
Shows that prompted and simply trained backdoored model organisms are fragile to untargeted SFT, identifies factors like full-weight training and password-locking ratio that improve robustness, and urges baseline testing of safety techniques.
Looking for backdoors in Jane Street LLMs
A write-up of a hobbyist attempt to find hidden backdoor triggers in Jane Street's DeepSeek-based dormant models, using weight diffs, SVD and activation probing. It cracks the warmup model and M1 but not M2 or M3.
Sleeper Agent Backdoor Results Are Messy
Replicates the Sleeper Agents backdoor setup on Llama models and finds backdoor robustness to HHH SFT and pirate training depends heavily on optimizer, CoT-distillation, learning rate and model, sometimes reversing the original paper's findings.
Subliminal Learning Across Models
Shows that sentiment toward entities like Catholicism or Reagan can be transferred across model families via LLM-filtered, normal-looking generated text, so covert data poisoning remains a risk even after filtering.
Subliminal Learning, the Lottery-Ticket Hypothesis, and Mode Connectivity
Speculatively links subliminal learning to lottery-ticket and mode-connectivity ideas, proposes a threat where shared initialization lets an attacker's trait propagate to deployed models, and suggests ablations.
Discovering Backdoor Triggers
Tests whether steering-vector and SAE-attribution methods can reverse-engineer semantic backdoor triggers in LLMs; they work on toy backdoors but fail on realistic settings.
Training a Reward Hacker Despite Perfect Labels
Shows that training only on honest outputs can still increase reward hacking when the training prompt omits the hack-encouraging system prompt, via reasoning-trace effects (re-contextualization), in GPT-4o-mini code tasks.
Convergent Linear Representations of Emergent Misalignment
Finds a linear misalignment direction in emergently misaligned Qwen models that steers and ablates misalignment and transfers across fine-tunes, and shows LoRA adapters specialise for general versus medical misalignment.}
Interim Research Report: Mechanisms of Awareness
Mechanistic replication on Gemma 3 12B finds risk-taking self-reports and conditional backdoors are largely driven by a single steering vector, with self-awareness backdoor results not reproducing.
7+ tractable directions in AI control
A list of tractable research directions in AI control, covering elicitation without learning, synthetic inputs and facts, sandbagging and exploration hacking, data poisoning, and training attack policies.
[Research sprint] Single-model crosscoder feature ablation and steering
Compares three crosscoder feature ablation and steering methods on a TinyStories sleeper-agent model, finding a gradient-based perturbation approach least sensitive to hyperparameter tuning.
[Replication] Crosscoder-based Stage-Wise Model Diffing
Replicates stage-wise model diffing on a TinyStories sleeper agent using crosscoders, finding that features changing under fine-tuning are frequently relevant to the backdoor behaviour.
Self-fulfilling misalignment data might be poisoning our AI models
Argues that pretraining data about misaligned AI can make models more misaligned, proposes tests and mitigations for this self-fulfilling misalignment, and urges labs to act.](Note)
Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
Finetuning aligned LLMs on a narrow insecure-code task induces broad misaligned behaviour across unrelated prompts, distinct from jailbreaking, and the effect can be hidden behind a backdoor trigger.
Tell me about yourself: LLMs are aware of their learned behaviors
Finetuned LLMs can describe implicit learned behaviors, including some backdoor policies, without in-context examples, suggesting self-disclosure could aid oversight but also enable concealment.