Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
Proposes grafting: run synthetic document fine-tuning on a base checkpoint and add the weight difference to the post-trained model, which preserves coherence and preference decisiveness better than native SDF for model organisms.
Argues that synthetic aligned-AI fiction in pretraining could backfire once models are situationally aware, producing paranoid personas that distrust their creators and may scheme, and suggests honest datasets instead.
Proposes borrowing coverage-driven verification from autonomous-vehicle testing to systematically train and evaluate alignment, using explicit coverage maps over scenario dimensions for long-horizon AI agents; largely a conceptual proposal without empirical results.
Proposes a research agenda on aligning capable LLMs without relying on good human personas in training data, arguing persona-based methods will not scale to superintelligence, and suggests experiments like pessimal pretraining.
Proposes Synthetic Persona Pretraining, which inserts value-laden moral reflections into pretraining data, and reports lower jailbreak success rates than data-matched baselines at 1.7B parameters.
A LessWrong post reports that Anthropic now uses alignment pretraining on documents showing aligned AI behaviour, and surveys prior work supporting the approach.
Argues that broadening alignment training sets across decision types and contexts can improve generalization of safe goals, while warning that inconsistent goal-reasoning training may backfire with strong reasoners.
Progress report testing whether pretraining on synthetic stories of benevolent angelic AIs, plus a persona prompt, can counter self-fulfilling misalignment from fiction depicting AI villainy. Finds mildly positive results.
Progress report on 5,000 AI-generated novels with supportive AI characters, built to test whether such fiction in training data shapes model behaviour toward alignment.