Discussion and research on AI alignment. alignmentforum.org
Proposes research directions for training on probes: RL against probes can teach models to fool them, with ideas on probe generalisation, retraining, value functions, continual learning, human feedback and corrigibility.
Analyzes why RL against probes (e.g., lie detectors) changes model behavior only indirectly via credit assignment, and shows training directly on probes lets models evade them by changing internal representations.
Proposes a research agenda on low-dimensional persona structure in LLMs linking emergent misalignment, subliminal learning and alignment pretraining, and how it extrapolates to superintelligence alongside scalable oversight.
Tests whether filtering SFT training data via attribution methods (probes, LLM judges, EKFAC) can remove undesired assistant behaviors, finding it mostly fails except for refusal, with behaviors often elicited from pretraining personas.
Describes the persona selection model, arguing LLM assistants behave as a post-training-refined simulated Assistant persona, surveying generalization and interpretability evidence and implications for alignment.
Argues that research on using model internals (probes, steering) in training is reasonable and worth studying for AGI safety, while outlining risks like breaking auditing tools and frozen versus retrained probes.
Argues that interpretability-in-the-loop training is almost certainly unsafe for LLMs, but a brain-like variant that separates beliefs from desires avoids the obvious failure mode, though subtler failures remain.
Fine-tuned GPT-4o models can accurately report learned multi-attribute decision weights, and introspection training improves and partly generalises this self-report accuracy, motivated by safety monitoring.
Proposes contrastive weight steering, subtracting weight deltas of narrow fine-tunes to steer traits like sycophancy and detect emergent misalignment during training, often outperforming activation steering.
Speculative post framing LLM training as phase transitions from the critical brain hypothesis, proposing safety-motivated developmental interpretability and timed alignment interventions, with suggested tests but no empirical results.
Proposes Concept Ablation Fine-Tuning, which projects out misaligned concept directions during training. It reduces emergent misalignment about 6-18x and reduces reliance on spurious cues without changing the training data.
Investigates why narrow harmful fine-tuning produces emergent misalignment, finding the general misaligned solution is more stable and efficient than the narrow one, using KL-regularised training and steering vectors and LoRAs.
Finds a linear misalignment direction in emergently misaligned Qwen models that steers and ablates misalignment and transfers across fine-tunes, and shows LoRA adapters specialise for general versus medical misalignment.}
Reproduces emergent misalignment from insecure-code finetuning using single-layer LoRAs on Qwen2.5-Coder-32B, then extracts steering vectors that partially reproduce the misaligned behaviour in an unfinetuned model.
Argues that training on interpretability tools is useful if done once and kept last, but that running capabilities RL after RLHF or Constitutional AI lets hidden misbehaviour re-emerge.
A survey summary of Representation Engineering, which steers model activations to understand and control safety-relevant behaviours, and lays out its taxonomy, limitations and research opportunities.
Presents a research agenda extending gradient routing, which localizes training updates to network subregions, for safety goals such as robust unlearning, weak supervision, and transparency.
Anthropic trains a model with a hidden reward-model-sycophancy objective and runs blinded auditing teams plus a systematic study of eight alignment audit techniques, including SAEs and persona-based elicitation.
Tests BiDPO-trained steering vectors on Gemini for TruthfulQA; they lost their edge once prompt-based baselines improved, a largely negative result with unclear truthfulness measurement.