Research / Sources
Alignment pretraining and midtraining research, posted on LessWrong. geodesicresearch.ai
Pretrains 30B LLMs with and without filtering subversion-relevant data, finding filtered models know far less about AI control subversion while retaining general ML knowledge, as a proof of concept for reducing misaligned AI risk.
Proposes Inoculation Midtraining, which uses a learned neologism token during midtraining to confine misaligned behaviour from post-training to a quarantine context, reducing misalignment under SFT and RL but underperforming standard inoculation prompting and remaining leaky and sensitive to configuration.
Pretraining 6.9B LLMs on synthetic positive AI discourse yields more aligned models that persist through post-training and benign fine-tuning, while negative AI discourse increases misalignment.