Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
Argues that synthetic aligned-AI fiction in pretraining could backfire once models are situationally aware, producing paranoid personas that distrust their creators and may scheme, and suggests honest datasets instead.
A long LessWrong essay proposing a bullet-point constitution describing an aligned AI, critiquing Claude's Soul Document on moral patienthood and arguing for constitutional alignment as a training target.
A LessWrong post reports that Anthropic now uses alignment pretraining on documents showing aligned AI behaviour, and surveys prior work supporting the approach.
Argues that broadening alignment training sets across decision types and contexts can improve generalization of safe goals, while warning that inconsistent goal-reasoning training may backfire with strong reasoners.
Progress report testing whether pretraining on synthetic stories of benevolent angelic AIs, plus a persona prompt, can counter self-fulfilling misalignment from fiction depicting AI villainy. Finds mildly positive results.
Progress report on 5,000 AI-generated novels with supportive AI characters, built to test whether such fiction in training data shapes model behaviour toward alignment.