Research / Sources
Alignment pretraining and midtraining research, posted on LessWrong. geodesicresearch.ai
Pretrains 30B LLMs with and without filtering subversion-relevant data, finding filtered models know far less about AI control subversion while retaining general ML knowledge, as a proof of concept for reducing misaligned AI risk.
Geodesic Research describes compute and model-access bottlenecks that independent AI safety organisations face versus frontier labs, and proposes philanthropic interventions such as shared compute pools and brokered agent access.
Proposes Inoculation Midtraining, which uses a learned neologism token during midtraining to confine misaligned behaviour from post-training to a quarantine context, reducing misalignment under SFT and RL but underperforming standard inoculation prompting and remaining leaky and sensitive to configuration.
Proposes studying 'proto-training gaming', where models begin reasoning about selection pressure during RL, and whether pre-RL alignment interventions can prevent adversarial misalignment from forming.
Argues that alignment interventions on pre-RL checkpoints (pretraining, midtraining, warm-start SFT) matter for preventing proto-training gaming and adversarial misalignment selected during production RL.
Announces Geodesic Research, an alignment org whose agenda is building robust alignment initialisations via midtraining and warm-start SFT so misalignment from capabilities RL is avoided.
Pretraining 6.9B LLMs on synthetic positive AI discourse yields more aligned models that persist through post-training and benign fine-tuning, while negative AI discourse increases misalignment.