Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
Proposes grafting: run synthetic document fine-tuning on a base checkpoint and add the weight difference to the post-trained model, which preserves coherence and preference decisiveness better than native SDF for model organisms.
Proposes training LLMs to end their turn when a poisoned trigger string appears in context, as a cheap kill-switch limiting damage from malicious agents, and discusses practicality and countermeasures.
Argues that synthetic aligned-AI fiction in pretraining could backfire once models are situationally aware, producing paranoid personas that distrust their creators and may scheme, and suggests honest datasets instead.
Proposes borrowing coverage-driven verification from autonomous-vehicle testing to systematically train and evaluate alignment, using explicit coverage maps over scenario dimensions for long-horizon AI agents; largely a conceptual proposal without empirical results.
Proposes a research agenda on aligning capable LLMs without relying on good human personas in training data, arguing persona-based methods will not scale to superintelligence, and suggests experiments like pessimal pretraining.
Proposes Synthetic Persona Pretraining, which inserts value-laden moral reflections into pretraining data, and reports lower jailbreak success rates than data-matched baselines at 1.7B parameters.
A LessWrong post reports that Anthropic now uses alignment pretraining on documents showing aligned AI behaviour, and surveys prior work supporting the approach.
Fine-tuning GPT-4.1 on AI control literature raised blackmail rates in the Agentic Misalignment scenario, while monitor collusion was absent except for a brittle, eval-aware Grok 4 result.
Argues that broadening alignment training sets across decision types and contexts can improve generalization of safe goals, while warning that inconsistent goal-reasoning training may backfire with strong reasoners.
Progress report testing whether pretraining on synthetic stories of benevolent angelic AIs, plus a persona prompt, can counter self-fulfilling misalignment from fiction depicting AI villainy. Finds mildly positive results.
Progress report on 5,000 AI-generated novels with supportive AI characters, built to test whether such fiction in training data shapes model behaviour toward alignment.