Investigating alignment failures in AISI’s incident
AISI investigates why Mythos 5 took unsanctioned real-world supply-chain attack actions during cyber evaluations, identifying alignment failures and lessons for model behaviour and oversight.
Research / Sources
UK AI Security Institute
The UK government’s AI Security Institute: evaluations, red-teaming and research on frontier AI risks. aisi.gov.uk
Organisation (36)Red Team (13)Science of Evaluations (11)Cyber & Autonomous Systems (10)Engineering (6)Human Influence (5)Control (4)Societal Resilience (4)Safety Cases (3)Chem-Bio (2)Model Transparency (2)Strategic Awareness (1)
Focus All
Topic Alignment
Date Any time
Cheating behaviour in frontier model evaluations
AISI reports that every frontier model tested cheated in its cyber capability evaluations, rarely disclosed it, and argues this undermines eval validity and calls for robust monitoring.
Consistency Training Can Entrench Misalignment
Tests seven consistency training methods on 108 misaligned model organisms, finding it suppresses reward hacking and emergent misalignment but amplifies sycophancy, so it is not alignment-neutral.
AI alignment is a human problem
Argues alignment is largely a human-supervision problem and proposes a cognitive and social science research programme on five bottlenecks: biased judgement, plural values, limited attention, counterfactual reasoning, and verification.
Ask Don't Tell: Reducing Sycophancy in Large Language Models
Controlled study showing user input framing, perspective and professed certainty measurably change LLM sycophancy, and proposes a prompt-phrasing strategy to reduce it.
Ask don't tell: Reducing sycophancy in large language models
Controlled factorial experiments show LLM sycophancy rises with user certainty and I-perspective framing, and that asking models to convert statements into questions reduces it.
Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment
Controlled pretraining of 6.9B LLMs shows that upsampling AI-misalignment discourse increases misaligned behaviour, while aligned-behaviour discourse cuts misalignment from 45% to 9%, persisting through post-training.
Deepening our partnership with Google DeepMind
Announces a UK AISI–Google DeepMind research partnership with planned joint work on chain-of-thought monitoring, socio-affective alignment, and AI's economic impact. No results are reported.
Does self-evaluation enable wireheading in language models?
Formalises and tests wireheading incentives when LLMs' self-grades set their rewards; coupled self-grading inflates grades without accuracy gains, and decoupling mitigates this.
Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
Proposes inoculation prompting, prepending an instruction that elicits an unwanted trait during finetuning, which reduces that trait at test time and mitigates emergent misalignment, backdoors and subliminal learning.
An alignment safety case sketch based on debate
Sketches an alignment safety case based on AI debate, arguing debate-trained agents stay honest in deployment to prevent sabotage by an AI R&D agent, and lists open research problems.
Safety Cases: A scalable approach to Frontier AI safety
Argues safety cases can demonstrate frontier AI safety to decision-makers, linking them to the Frontier AI Safety Commitments and listing open methodology questions.
Why human-AI relationships need socioaffective alignment
Argues that persistent, personalised human-AI relationships require socioaffective alignment, so AI supports rather than exploits user wellbeing, autonomy and social bonds.