Enhancing Model Safety through Pretraining Data Filtering
Filters CBRN weapons information from pretraining data, then pretrains models from scratch, cutting accuracy on a harmful-capabilities eval by 33% relative to baseline while preserving benign capabilities.