Research

Alignment Forum

Discussion and research on AI alignment. alignmentforum.org

AI (574)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)

1 item

4 Nov 2025BlogAlignment, Jailbreaks

GDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash

TurnTrout and Rohin Shah

Proposes consistency training, which teaches models to ignore irrelevant user biases and jailbreak wrappers. It reduces sycophancy and jailbreak success in Gemma and Gemini 2.5 Flash, with BCT strongest against jailbreaks.

Jailbreaking (AIs)AI

1 item

7 Jan 2025BlogJailbreaks, Alignment

Role embeddings: making authorship more salient to LLMs

Nina Panickssery and Christopher Ackerman

Proposes role embeddings that tag every token with its speaker role to make Llama 3 more robust to many-shot jailbreaks and prompt injection, flattening the attack power law without hurting general capability in initial experiments.

Jailbreaking (AIs)AI

Research

Filters

Focus

Date

Sort

Alignment Forum

GDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash

Role embeddings: making authorship more salient to LLMs

Research

Tuesday 4 November

GDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash

Tuesday 7 January

Role embeddings: making authorship more salient to LLMs