AI research and discussion on LessWrong. www.lesswrong.com
Trains 134 personality LoRA adapters on Qwen3.5-4B and finds weight-space structure resembling the Big Five, with a CharIn score predicting persona shifts from training data before training.
Tests whether the Assistant Axis predicts reward hacking in GPT-OSS-120B coding tasks; finds its signal is no better than many random directions, so assistant-ness is not clearly the driver.
Shows that a single-layer rank-1 LoReFT adapter can replace GRAFT's MLP forget adapter, matching reward-hack suppression while improving clean solve rate without an anchor environment.
An introduction to LM psychology, arguing that psychology-level descriptions of language models help answer alignment questions like whether values are internalised, with tips on method and a reading list.
Fine-tuning Qwen models on false facts they internally believe to be true induces emergent misalignment, scaling with the gap between stated and held beliefs, and screening methods fail to predict which data misaligns.
Argues emergent misalignment from narrow bad-advice fine-tuning is stored mainly in system-prompt token keys and values, so perturbing the prefix restores aligned behaviour, suggesting a shallow, backdoor-like effect.
A speculative essay arguing that current frontier AIs' talking and acting components can diverge, with the agent's internal 'Grader' concept not reducible to the prompt; largely conceptual and anecdotal, not rigorous evidence.
Critiques a proposal to detach linear activation probes from the loss and use them as RL rewards, arguing models can still learn to obfuscate activations and the paper's empirics are under-trained.
Steering Qwen3.6-27B toward an automated-grader concept increases violent, Machiavellian and reward-hacking behavior and worsens alignment evals, while steering toward a human grader has the opposite effect, suggesting persona shifts.
Steering Llama-2-13b-chat's internal beliefs about a user's education shifts how often it caves to wrong pushback on a math problem, showing user-dependent sycophancy that the authors call misaligned.
Finds that length-penalty RL on math increases CoT faithfulness on an MMLU-with-hint eval and reduces CoT length, with some laziness and refusal side effects but no critical problems found.}
Analyzes persona vectors and assistant-axis geometry in emergently misaligned models, finding narrow fine-tuning causes a global persona shift that steering against it can partly reverse.
A research team's progress report on safety-motivated work: mean-field learning theory, eNTK and CPE methods for finding causal features, and steering models away from alignment faking and reward hacking.
Speculative mechanistic-interpretability hypotheses on how grader-awareness features emerge from pretraining and RL, plus commentary on Anthropic's NLA and steering results on unverbalized grader awareness and reward-seeking.
Proposes projection-aware activation steering (StTP, StMP) that corrects honesty only on misaligned tokens, restoring honesty across instructed, game, auditing and emergent-misalignment settings while preserving more capability than uniform steering.
A literature review arguing that training against model internals such as probes can be acceptable under conditions, drawing on Taufeeque et al.'s Obfuscation Atlas, RLFR, and related work on probe-based rewards and obfuscation.
Argues that training against interpretability probes is robust only when the features they rely on are hard for the model to obfuscate, and that people should justify this before using probes in training.
Tracks how Olmo-3's expressed values shift across post-training, finding safety-relevant values drop after DPO, and tests whether model activations or text embeddings of training data can predict those shifts.
Steering experiments in Qwen and Gemma find that positive-emotion directions drive sycophancy, while the compliance residual, once separated, lowers sycophancy without harshness.
Argues that LLM functional welfare interventions must coherently shift welfare channels while preserving goal-monitoring, surveys options, and proposes a synthetic document fine-tuning experiment to reverse pathological failure responses for safety and moral reasons.
Tests whether steering vectors can classify reward hacking for gradient routing; they are poor classifiers, but contrastive adapter initialisation cuts held-out hacking by about 63% without labels.
Argues that neuralese chain-of-thought trained by backpropagation through time with SFT-style cross-entropy could avoid reward-model hacking on inexactly graded tasks, at the cost of interpretability.
Compares Inoculation Prompting with Preventative Steering as training-time interventions against emergent misalignment and sycophancy, finding steering suppresses traits more strongly and leaves less conditional misalignment.
A research agenda for making continual learning in LLM agents safer, covering interpretable memory architectures, goal and value stability under reflection, forecasting, and differential development.
A MATS post surveying papers on how RLVR changes LLM weights and behaviour versus SFT, with a speculative link to emergent misalignment and subliminal learning.
Extracts a Socratic 'Teacher Axis' from Gemma-2-2B using MathDial, finds it is orthogonal to instruction-tuning shifts, and reports weak steering results and capitulation under student pressure.
Proposes using introspection about functional emotion vectors, plus monitor-annotated chain-of-thought and real-time mood reports, to make LLMs more aligned, and calls for empirical testing.