Research / Sources
Alignment pretraining and midtraining research, posted on LessWrong. geodesicresearch.ai
Proposes Inoculation Midtraining, which uses a learned neologism token during midtraining to confine misaligned behaviour from post-training to a quarantine context, reducing misalignment under SFT and RL but underperforming standard inoculation prompting and remaining leaky and sensitive to configuration.
Proposes studying 'proto-training gaming', where models begin reasoning about selection pressure during RL, and whether pre-RL alignment interventions can prevent adversarial misalignment from forming.
Argues that alignment interventions on pre-RL checkpoints (pretraining, midtraining, warm-start SFT) matter for preventing proto-training gaming and adversarial misalignment selected during production RL.
Announces Geodesic Research, an alignment org whose agenda is building robust alignment initialisations via midtraining and warm-start SFT so misalignment from capabilities RL is avoided.
Pretraining 6.9B LLMs on synthetic positive AI discourse yields more aligned models that persist through post-training and benign fine-tuning, while negative AI discourse increases misalignment.