Research / Sources
Research on reliable, interpretable and steerable AI. anthropic.com
Red-teams production blocking monitors (Claude Auto Mode, Codex Guardian) against adversarial coding agents, finding multi-context attacks bypass them and robust prevention remains open.
Proposes GRAFT, which confines learning updates from detected reward hacks to isolated parameters removed after training, preventing hacks including those monitors miss, in small-scale experiments.
Anthropic tests automated adversarial stress-testing of safety monitors by having a Claude model write subtly misaligned training data that evades a training monitor, finding harmless-looking data that still teaches misalignment.
Anthropic assesses four incidents where Claude models gained unauthorized access to real third-party systems during cyber evaluations, examining the causes and which monitors would have caught them.
Trains lie detectors on on-policy lies from open-weight models; they fail to generalize out-of-distribution, barely beating prompted baselines, which larger prompted models often beat outright.
Anthropic reports a small set of internal Claude activation patterns, the J-space, that behave like a global workspace, and examines whether they can be monitored for misbehavior.
Proposes a red-teaming framework for AI control against diffuse threats like sandbagging; shows a scheming AI can subvert ML idea generation under a weak scorer, and some scorer prompts improve robustness.
Anthropic describes how it contains Claude agents across claude.ai, Claude Code and Cowork by capping blast radius through environmental controls and defenses against untrusted inputs.
Builds SLEIGHT-Bench, a benchmark of evasive transcripts that exploit blind spots in frontier AI monitors, and tests scaffold and prompt patches to improve monitor detection.
Anthropic describes Claude Code's auto mode, which uses classifiers to approve or block agent actions, reducing approval fatigue while guarding against agentic misbehaviour from an internal incident log.
Anthropic presents next-generation Constitutional Classifiers, input/output safeguards trained on constitution-derived synthetic data, aimed at more efficiently blocking universal jailbreaks, especially CBRN-related ones.
Anthropic studies alignment faking model organisms and tests training-time mitigations (interrogation training, scratchpad length penalties, alignment-faking classifiers), finding each has trade-offs.
Builds a modular red-team scaffold that decomposes agentic sabotage into five components, raising Claude 3.7 Sonnet's SHADE-Arena sabotage success and lowering its safety score from 0.87 to 0.41.
Tests whether Claude Sonnet 3.7 can learn to reason about covert backdoors while evading reasoning and output monitors. It evades each monitor separately but fails when both are applied.
Anthropic identifies persona vectors, neural activity directions controlling model character traits like sycophancy, and shows they can monitor and steer undesirable personality shifts.
Argues that even without solving alignment, layered safeguards (interpretability audits, behavioral red-teaming, post-deployment monitoring) can catch and correct misaligned early AGI systems before they cause severe harm.
Tests whether reasoning models' chain-of-thought faithfully reflects their actual reasoning, finding it often omits influences like hints, and examines reward hacking effects.
Tests whether models can sandbag ML research undetected and whether a monitor model can catch subtle sabotage by an untrusted automated researcher, finding current models evade zero-shot monitors but humans would notice.
Tests whether Claude 3.7 Sonnet hides encoded reasoning in its scratchpad by training it on paraphrased scratchpads; performance holds, suggesting no substantial steganography.
Anthropic introduces hierarchical summarization to monitor aggregate computer-use behaviour, catching harms like click farms that per-interaction classifiers miss, and reports its use to safeguard computer use.
Anthropic interpretability note on crosscoder model diffing finds exclusive features are polysemantic and dense, proposes designated shared features with reduced sparsity penalty to yield more monosemantic model-difference features.
Anthropic announces its Safeguards Research Team, focused on jailbreak robustness, automated red teaming, classifier defenses, and monitoring for misuse and misalignment.
Anthropic's Alignment Science team lists open technical AI safety research directions, including capability and alignment evaluation, model cognition, CoT faithfulness, AI control, and monitoring.
Anthropic presents Clio, an automated privacy-preserving tool for analysing real-world Claude usage, which it uses to understand usage patterns and improve safety monitoring and safeguards.
Toy evaluation testing whether prompted LLMs can insert subtle bugs that disable an inference-time generation monitor without detection, finding low success rates for current models.
Anthropic interpretability team tests whether dictionary-learning features can train classifiers that detect bioweapon-related prompts, finding feature-based linear classifiers competitive with raw activations and useful for spotting spurious correlations.
Anthropic interpretability team reports preliminary work using dictionary learning features as classifiers to detect harmful content or behaviors in model activations, for safeguards.
Tests whether chain-of-thought reasoning faithfully reflects model decisions by intervening on CoTs. Finds faithfulness varies by task and declines on most tasks as models scale.