Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Proposes consistency training, which teaches models to ignore irrelevant user biases and jailbreak wrappers. It reduces sycophancy and jailbreak success in Gemma and Gemini 2.5 Flash, with BCT strongest against jailbreaks.
Shows that fine-tuning via a jailbreak-tuning attack strips safety guardrails from DeepSeek R1 and fine-tunable GPT-4o, Claude 3 Haiku and Gemini models, arguing current safeguards are illusory and calling for evil-twin evaluation.
Proposes role embeddings that tag every token with its speaker role to make Llama 3 more robust to many-shot jailbreaks and prompt injection, flattening the attack power law without hurting general capability in initial experiments.