Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Proposes Power Steering, which finds LLM steering vectors from layer-to-layer Jacobian singular vectors, matching MELBO on refusal and corrigibility steering while being far cheaper to compute.
Proposes consistency training, which teaches models to ignore irrelevant user biases and jailbreak wrappers. It reduces sycophancy and jailbreak success in Gemma and Gemini 2.5 Flash, with BCT strongest against jailbreaks.
Fine-tuned and prompted LLMs gain reasoning uplift only from very simple ciphers, suggesting encoded chain-of-thought is hard for current models and bearing on CoT monitorability and cipher-based jailbreaks.
Benchmarks commercial jailbreak-detection supervisors (BELLS), finding them weak and that simply prompting frontier LLMs to classify harmfulness works better, though those models often still answer harmful queries they flag.
Shows layered defenses for LLMs against CBRN misuse can be broken by a staged attack (STACK) reaching 71% success, while conventional attacks got 0%, and proposes hardening recommendations.
Shows that distilling an unlearned model into a fresh random-initialized student makes hazardous capabilities robust to relearning attacks, and proposes UNDO to trade compute for robustness.
Tests whether an LLM prompt evaluator can block bioweapons-related questions from the ProtocolQA benchmark, finding blocking rates depend strongly on subtle wording of the evaluator's task.
Shows that fine-tuning via a jailbreak-tuning attack strips safety guardrails from DeepSeek R1 and fine-tunable GPT-4o, Claude 3 Haiku and Gemini models, arguing current safeguards are illusory and calling for evil-twin evaluation.
Proposes DATDP, a defense that repeatedly uses an evaluation LLM to screen prompts, blocking nearly all Best-of-N augmented jailbreak attempts, even with small models.
Proposes role embeddings that tag every token with its speaker role to make Llama 3 more robust to many-shot jailbreaks and prompt injection, flattening the attack power law without hurting general capability in initial experiments.