Discussion and research on AI alignment. alignmentforum.org
Proposes research directions for training on probes: RL against probes can teach models to fool them, with ideas on probe generalisation, retraining, value functions, continual learning, human feedback and corrigibility.
Analyzes why RL against probes (e.g., lie detectors) changes model behavior only indirectly via credit assignment, and shows training directly on probes lets models evade them by changing internal representations.
Proposes a research agenda on low-dimensional persona structure in LLMs linking emergent misalignment, subliminal learning and alignment pretraining, and how it extrapolates to superintelligence alongside scalable oversight.
Tests whether filtering SFT training data via attribution methods (probes, LLM judges, EKFAC) can remove undesired assistant behaviors, finding it mostly fails except for refusal, with behaviors often elicited from pretraining personas.
Describes the persona selection model, arguing LLM assistants behave as a post-training-refined simulated Assistant persona, surveying generalization and interpretability evidence and implications for alignment.
Argues that research on using model internals (probes, steering) in training is reasonable and worth studying for AGI safety, while outlining risks like breaking auditing tools and frozen versus retrained probes.
Argues that interpretability-in-the-loop training is almost certainly unsafe for LLMs, but a brain-like variant that separates beliefs from desires avoids the obvious failure mode, though subtler failures remain.
Fine-tuned GPT-4o models can accurately report learned multi-attribute decision weights, and introspection training improves and partly generalises this self-report accuracy, motivated by safety monitoring.
Proposes contrastive weight steering, subtracting weight deltas of narrow fine-tunes to steer traits like sycophancy and detect emergent misalignment during training, often outperforming activation steering.