Discussion and research on AI alignment. alignmentforum.org
Tests whether filtering SFT training data via attribution methods (probes, LLM judges, EKFAC) can remove undesired assistant behaviors, finding it mostly fails except for refusal, with behaviors often elicited from pretraining personas.
Argues that research on using model internals (probes, steering) in training is reasonable and worth studying for AGI safety, while outlining risks like breaking auditing tools and frozen versus retrained probes.
Argues that interpretability-in-the-loop training is almost certainly unsafe for LLMs, but a brain-like variant that separates beliefs from desires avoids the obvious failure mode, though subtler failures remain.
Fine-tuned GPT-4o models can accurately report learned multi-attribute decision weights, and introspection training improves and partly generalises this self-report accuracy, motivated by safety monitoring.
Finds a linear misalignment direction in emergently misaligned Qwen models that steers and ablates misalignment and transfers across fine-tunes, and shows LoRA adapters specialise for general versus medical misalignment.}
Argues that training on interpretability tools is useful if done once and kept last, but that running capabilities RL after RLHF or Constitutional AI lets hidden misbehaviour re-emerge.
A survey summary of Representation Engineering, which steers model activations to understand and control safety-relevant behaviours, and lays out its taxonomy, limitations and research opportunities.
Anthropic trains a model with a hidden reward-model-sycophancy objective and runs blinded auditing teams plus a systematic study of eight alignment audit techniques, including SAEs and persona-based elicitation.