Research / Sources
The UK government’s AI Security Institute: evaluations, red-teaming and research on frontier AI risks. aisi.gov.uk
Tests whether filtering viral proteins from ESM-2 pretraining data selectively degrades hazardous capability; finds it moderately effective but recoverable by cheap fine-tuning.
Tests whether filtering pretraining data can safeguard open-weight protein models against dual-use biological capabilities; finds it selective and effective but not durable against fine-tuning.
Pretraining from scratch on data filtered of dual-use biothreat text yields open-weight 6.9B models resisting up to 10,000 steps of adversarial fine-tuning, though filtered models can still use in-context information.
Critiques current LLM unlearning evaluations, finding they inject new information, vary across tasks and rely on spurious correlations, and proposes minimal-injection and downstream-awareness principles.
Proposes model tampering attacks on latent activations and weights as a stronger way to evaluate hazardous LLM capabilities, finding unlearned capabilities often return within 16 fine-tuning steps.