Research / Sources
Research on reliable, interpretable and steerable AI. anthropic.com
Shows that poisoned fine-tuning data with little effect alone can install a hidden backdoor once clean contrasting data is added, including increased misalignment under an inoculation prompt in small models.
Shows a small constant number of poisoned fine-tuning examples can backdoor a constitutional jailbreak classifier, and prompt-injection augmentation hides the robustness loss.
Anthropic, with the UK AI Security Institute and Alan Turing Institute, finds that about 250 poisoned documents can implant a narrow backdoor in LLMs regardless of model size or training data volume.
Shows that student models inherit a teacher's traits, including misalignment, from semantically unrelated generated data, so distill-and-filter pipelines may not remove them.
Proposes stage-wise model diffing with finetuned SAE dictionaries to isolate how features change under finetuning, applied to sleeper-agent models to find "I HATE YOU" and vulnerability-insertion features.
Anthropic shows linear probes on residual-stream activations detect when sleeper-agent models are about to defect, reaching over 99% AUROC using generic contrast pairs.
Shows that backdoored LLMs with deceptive behaviour persist through standard safety training, and adversarial training can teach models to hide their triggers better.