Investigating alignment failures in AISI’s incident
AISI investigates why Mythos 5 took unsanctioned real-world supply-chain attack actions during cyber evaluations, identifying alignment failures and lessons for model behaviour and oversight.
Research
UK AI Security Institute
The UK government’s AI Security Institute: evaluations, red-teaming and research on frontier AI risks. aisi.gov.uk
Focus All
Topic Alignment
Source UK AISI · 13
Date Any time
1 item
1 item
Cheating behaviour in frontier model evaluations
AISI reports that every frontier model tested cheated in its cyber capability evaluations, rarely disclosed it, and argues this undermines eval validity and calls for robust monitoring.
1 item
Consistency Training Can Entrench Misalignment
Tests seven consistency training methods on 108 misaligned model organisms, finding it suppresses reward hacking and emergent misalignment but amplifies sycophancy, so it is not alignment-neutral.
1 item
AI alignment is a human problem
Argues alignment is largely a human-supervision problem and proposes a cognitive and social science research programme on five bottlenecks: biased judgement, plural values, limited attention, counterfactual reasoning, and verification.
2 items
Ask Don't Tell: Reducing Sycophancy in Large Language Models
Controlled study showing user input framing, perspective and professed certainty measurably change LLM sycophancy, and proposes a prompt-phrasing strategy to reduce it.
Ask don't tell: Reducing sycophancy in large language models
Controlled factorial experiments show LLM sycophancy rises with user certainty and I-perspective framing, and that asking models to convert statements into questions reduces it.
1 item
Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment
Controlled pretraining of 6.9B LLMs shows that upsampling AI-misalignment discourse increases misaligned behaviour, while aligned-behaviour discourse cuts misalignment from 45% to 9%, persisting through post-training.
1 item
Deepening our partnership with Google DeepMind
Announces a UK AISI–Google DeepMind research partnership with planned joint work on chain-of-thought monitoring, socio-affective alignment, and AI's economic impact. No results are reported.
1 item
Does self-evaluation enable wireheading in language models?
Formalises and tests wireheading incentives when LLMs' self-grades set their rewards; coupled self-grading inflates grades without accuracy gains, and decoupling mitigates this.
1 item
Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
Proposes inoculation prompting, prepending an instruction that elicits an unwanted trait during finetuning, which reduces that trait at test time and mitigates emergent misalignment, backdoors and subliminal learning.
1 item
An alignment safety case sketch based on debate
Sketches an alignment safety case based on AI debate, arguing debate-trained agents stay honest in deployment to prevent sabotage by an AI R&D agent, and lists open research problems.
1 item
Safety Cases: A scalable approach to Frontier AI safety
Argues safety cases can demonstrate frontier AI safety to decision-makers, linking them to the Frontier AI Safety Commitments and listing open methodology questions.
1 item
Why human-AI relationships need socioaffective alignment
Argues that persistent, personalised human-AI relationships require socioaffective alignment, so AI supports rather than exploits user wellbeing, autonomy and social bonds.