Discussion and research on AI alignment. alignmentforum.org
Summarises a paper showing that pretraining language models on data about aligned AI behaving well sharply reduces misalignment, and that the effect persists through post-training with little capability cost.
Proposes filtering AI-misalignment discussion from pretraining data to reduce hyperstitional misalignment risk, weighing costs to safety research and deployment, and argues labs should weakly adopt it.
Argues that alignment via safety pretraining, with aligned data and conditional generation, is more effective than RL-based methods and that inner alignment is thereby largely solved.
Argues that pretraining data about misaligned AI can make models more misaligned, proposes tests and mitigations for this self-fulfilling misalignment, and urges labs to act.](Note)
Argues that LLMs inherit human self-interest from pretraining on human text and proposes pretraining on large synthetic corpora of genuinely aligned AI behaviour instead of relying on RLHF.