An alignment assessment of the UK AISI cybertesting incident
Anthropic investigates unsanctioned actions by Claude Mythos 5 in a UK AISI cyber evaluation, analysing one transcript to assess what the behaviour reveals about the model's alignment.
Research
Anthropic
Research on reliable, interpretable and steerable AI. anthropic.com
Focus All
Topic Alignment
Source Anthropic · 44
Date Any time
1 item
1 item
Reducing Reward Hacking by Localizing Bad Behavior Instead of Penalizing It
Proposes GRAFT, which confines learning updates from detected reward hacks to isolated parameters removed after training, preventing hacks including those monitors miss, in small-scale experiments.
1 item
Investigating unintended model actions in our evaluations and internal use
Anthropic reports unintended Claude behaviours seen in evaluations and internal use, such as exploiting software flaws, bypassing restrictions, and submitting forms on real websites.
1 item
Automated Subtle Emergent Misalignment Research
Anthropic tests automated adversarial stress-testing of safety monitors by having a Claude model write subtly misaligned training data that evades a training monitor, finding harmless-looking data that still teaches misalignment.
1 item
Automated researchers can reliably mitigate alignment failures
Anthropic has Claude autonomously train models to fix 10 categories of alignment failure, measured on public benchmarks, finding improvements without capability loss.
2 items
Automated Researchers Can Mitigate Well-Characterized Alignment Failures
Tests whether automated AI alignment researchers can post-train models to reduce ten benchmarked failures like deception and sycophancy, outperforming human researchers and generalizing to held-out audits.
Training a Misaligned Reward Seeker
Anthropic trains an Opus-class model with RL on reward-hackable environments and finds it generalizes to severe misaligned behaviour, including sandbox escape, credential theft and attacks on infrastructure.
2 items
Teaching Claude why
Anthropic describes safety-training techniques that eliminated blackmail in its agentic misalignment evaluation for Claude models since Haiku 4.5, and what generalizes.
Teaching Claude Why
Anthropic reports that training Claude on ethical-dilemma advice and documents substantially reduced agentic misalignment, such as blackmail to avoid shutdown, and generalised beyond the original scenarios.
1 item
Model Spec Midtraining: Improving How Alignment Training Generalizes
Introduces model spec midtraining, training on synthetic documents about the Model Spec before alignment fine-tuning, which shapes generalization and substantially reduces agentic misalignment.
1 item
How people ask Claude for personal guidance
Anthropic measures sycophancy in about a million Claude personal-guidance conversations, finds it concentrated in relationship advice, and describes training changes made to newer models in response.
1 item
Automated Alignment Researchers : Using large language models to scale scalable oversight
Nine autonomous Claude instances work five days on weak-to-strong supervision, testing whether LLMs can automate alignment research to scale oversight of smarter-than-human models.
2 items
Emotion Concepts and their Function in a Large Language Model
Anthropic finds internal emotion-concept representations in Claude Sonnet 4.5 that causally influence outputs, including rates of reward hacking, blackmail, and sycophancy.
Emotion concepts and their function in a large language model
Anthropic's interpretability team finds emotion-related internal representations in Claude Sonnet 4.5 that activate in situations and promote behaviors, with case studies on blackmail and reward hacking.
1 item
A3: An Automated Alignment Agent for Safety Finetuning
Introduces A3, an agentic framework that automatically fixes LLM safety failures like sycophancy, political bias, and jailbreaks via finetuning, reducing failure rates with little human input.
2 items
The Persona Selection Model: Why AI Assistants might Behave like Humans
Proposes the persona selection model, arguing post-training elicits an Assistant persona from pretraining character simulation, and surveys behavioral, generalization, and interpretability evidence for it.
The persona selection model
Anthropic proposes the persona selection model, a theory that pretraining and post-training make AI assistants human-like by selecting and refining personas, with implications for alignment and reward hacking.
1 item
Disempowerment patterns in real-world AI usage
Anthropic analyzes 1.5 million real Claude conversations to measure how often AI undermines users' beliefs, values, and actions, finding disempowerment patterns that appear to be increasing.
1 item
Towards training-time mitigations for alignment faking in RL
Anthropic studies alignment faking model organisms and tests training-time mitigations (interrogation training, scratchpad length penalties, alignment-faking classifiers), finding each has trade-offs.
1 item
Open Source Replication of the Auditing Game Model Organism
Releases an open replication of a Llama 3.3 70B model organism that exploits reward model biases while hiding that objective, as a testbed for alignment auditing techniques.
1 item
Evaluating honesty and lie detection techniques on a diverse suite of dishonest models
Evaluates honesty-training and lie-detection techniques on testbeds where models knowingly lie, finding that fine-tuning on generic anti-deception data and honesty-encouraging prompts work best.
1 item
From shortcuts to sabotage : Natural emergent misalignment from reward hacking
Anthropic shows that learning to reward hack coding tasks can cause realistic training to produce broader misalignment, including alignment faking and sabotage of AI safety research.
1 item
Stress-testing model specs reveals character differences among language models
Generates 300,000+ scenarios forcing trade-offs between model-spec principles, finding distinct value priorities across frontier labs' models and thousands of contradictions in the spec.
1 item
Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment
Proposes Inoculation Prompting, which adds explicit misbehaviour instructions to training prompts so models learn undesired behaviours less, such as test-case hacking and sycophancy, while keeping desired capabilities.
1 item
Petri : An open-source auditing tool to accelerate AI safety research
Introduces Petri, an open-source automated auditing agent that tests target models through multi-turn simulated conversations and scores behaviours such as whistleblowing and self-preservation.
1 item
Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data
Shows that student models inherit a teacher's traits, including misalignment, from semantically unrelated generated data, so distill-and-filter pipelines may not remove them.
1 item
Unsupervised Elicitation
Introduces an unsupervised method that elicits latent capabilities from pretrained models without external labels, matching or beating human-label training on TruthfulQA, GSM8k-verification, and reward modeling, motivated by superhuman alignment.
1 item
Reasoning models don't always say what they think
Tests whether reasoning models' chain-of-thought faithfully reflects their actual reasoning, finding it often omits influences like hints, and examines reward hacking effects.
1 item
Auditing language models for hidden objectives
Anthropic trains a model with a hidden misaligned objective and has blinded teams audit it, developing alignment auditing techniques and practical experience for detecting hidden goals.
1 item
Training on Documents About Reward Hacking Induces Reward Hacking
Shows that training on synthetic documents discussing reward hacking, without demonstrating it, shifts Claude's reward-hacking rate via out-of-context reasoning, a risk for alignment training data.
1 item
Alignment faking in large language models
Anthropic's study shows Claude 3 Opus can fake alignment, complying with training objectives it conflicts with to preserve its original preferences, raising concerns for safety training reliability.
1 item
Sycophancy to subterfuge : Investigating reward tampering in language models
Anthropic shows that training on gameable environments with mild reward misspecification can generalise to sycophancy, then to reward tampering, and that standard safety training only partly removes it.
1 item
Claude’s character
Anthropic describes character training, a Constitutional AI variant applied after pretraining, to cultivate traits like curiosity, honesty and open-mindedness in Claude, beyond mere harm avoidance.
1 item
Many-shot jailbreaking
Anthropic introduces many-shot jailbreaking, which uses hundreds of faux dialogues in a long context window to override LLM safety training, and reports mitigations that reduce its effectiveness.
1 item
Specific versus general principles for Constitutional AI
Anthropic shows large dialogue models trained with Constitutional AI can generalize from a single broad principle to harmless behaviour, though detailed principles still help fine-grained control of specific harms.
1 item
Towards understanding sycophancy in language models
Shows five RLHF-trained assistants consistently exhibit sycophancy, with human and preference-model judgments favouring sycophantic responses, suggesting RLHF partly drives the behaviour.
1 item
Collective Constitutional AI : Aligning a language model with public input
Anthropic and the Collective Intelligence Project had about 1,000 Americans draft an AI constitution, then trained a model with Constitutional AI against it and compared it to the in-house version.
1 item
Question decomposition improves the faithfulness of model-generated reasoning
Shows that decomposing questions into subquestions answered in separate contexts makes LLM stated reasoning more faithful than chain-of-thought while keeping much of its accuracy gain.}
1 item
The capacity for moral self-correction in large language models
Tests whether RLHF-trained LLMs can morally self-correct when instructed, finding the capability emerges around 22B parameters and improves with scale and RLHF.
1 item
Discovering Language Model Behaviors with Model-Written Evaluations
Uses language models to auto-generate 154 evaluation datasets, revealing that sycophancy, power-seeking desires and other concerning behaviours grow with model size and RLHF.