How AI agents shape experts’ judgment calls
Randomized trial showing expert analysts using a Claude Code agent deferred to the agent's randomly favored option 86% of the time, raising concerns for human oversight of AI-assisted work.
Research / Sources
Anthropic
Research on reliable, interpretable and steerable AI. anthropic.com
Focus All
Topic Oversight
Date Any time
Training Alignment Auditors via Reinforcement Learning
Trains an LLM alignment auditor with RL to find planted hidden behaviours; pairwise rewards resist reward hacking, and a small trained model matches a much larger one.
Automated Subtle Emergent Misalignment Research
Anthropic tests automated adversarial stress-testing of safety monitors by having a Claude model write subtly misaligned training data that evades a training monitor, finding harmless-looking data that still teaches misalignment.
TASTE: Can AI Models Judge AI Safety Research Proposals?
Introduces TASTE, a benchmark scoring whether models can judge pairs of AI safety research proposals against experienced researchers' preferences; models underperform humans (60% vs. 77% human agreement).
Automated researchers can reliably mitigate alignment failures
Anthropic has Claude autonomously train models to fix 10 categories of alignment failure, measured on public benchmarks, finding improvements without capability loss.
Automated Researchers Can Mitigate Well-Characterized Alignment Failures
Tests whether automated AI alignment researchers can post-train models to reduce ten benchmarked failures like deception and sycophancy, outperforming human researchers and generalizing to held-out audits.
Automated Alignment Researchers : Using large language models to scale scalable oversight
Nine autonomous Claude instances work five days on weak-to-strong supervision, testing whether LLMs can automate alignment research to scale oversight of smarter-than-human models.
Automated Weak-to-Strong Researcher
Anthropic builds Claude-powered agents that autonomously propose and run experiments on weak-to-strong supervision, outperforming human researchers on recovering strong-model performance.
3 Challenges and 2 Hopes for the Safety of Unsupervised Elicitation
Stress-tests unsupervised elicitation and easy-to-hard generalization methods on three realistic challenges (salient confounds, imbalanced data, uncertain points), finding that proposed ensembling and combined approaches still fail to reliably handle them.
Building and evaluating alignment auditing agents
Anthropic builds LLM agents that autonomously run alignment audits, uncovering hidden goals, building behavioral evals, and surfacing concerning behaviors in models with planted issues.
Unsupervised Elicitation
Introduces an unsupervised method that elicits latent capabilities from pretrained models without external labels, matching or beating human-label training on TruthfulQA, GSM8k-verification, and reward modeling, motivated by superhuman alignment.
Constitutional AI : Harmlessness from AI feedback
Introduces Constitutional AI, training a harmless assistant via self-critique, revision, and RL from AI feedback guided by a list of principles, with far fewer human labels.
Measuring progress on scalable oversight for large language models
Anthropic proposes an empirical design for scalable oversight and shows humans assisted by an unreliable LLM outperform both the model alone and unaided humans on MMLU and QuALITY.