AI research and discussion on LessWrong. www.lesswrong.com
Trains 134 personality LoRA adapters on Qwen3.5-4B and finds weight-space structure resembling the Big Five, with a CharIn score predicting persona shifts from training data before training.
Tests whether the Assistant Axis predicts reward hacking in GPT-OSS-120B coding tasks; finds its signal is no better than many random directions, so assistant-ness is not clearly the driver.
Shows that a single-layer rank-1 LoReFT adapter can replace GRAFT's MLP forget adapter, matching reward-hack suppression while improving clean solve rate without an anchor environment.
An introduction to LM psychology, arguing that psychology-level descriptions of language models help answer alignment questions like whether values are internalised, with tips on method and a reading list.
Fine-tuning Qwen models on false facts they internally believe to be true induces emergent misalignment, scaling with the gap between stated and held beliefs, and screening methods fail to predict which data misaligns.
Argues emergent misalignment from narrow bad-advice fine-tuning is stored mainly in system-prompt token keys and values, so perturbing the prefix restores aligned behaviour, suggesting a shallow, backdoor-like effect.
A speculative essay arguing that current frontier AIs' talking and acting components can diverge, with the agent's internal 'Grader' concept not reducible to the prompt; largely conceptual and anecdotal, not rigorous evidence.
Critiques a proposal to detach linear activation probes from the loss and use them as RL rewards, arguing models can still learn to obfuscate activations and the paper's empirics are under-trained.
Steering Qwen3.6-27B toward an automated-grader concept increases violent, Machiavellian and reward-hacking behavior and worsens alignment evals, while steering toward a human grader has the opposite effect, suggesting persona shifts.
Steering Llama-2-13b-chat's internal beliefs about a user's education shifts how often it caves to wrong pushback on a math problem, showing user-dependent sycophancy that the authors call misaligned.
Finds that length-penalty RL on math increases CoT faithfulness on an MMLU-with-hint eval and reduces CoT length, with some laziness and refusal side effects but no critical problems found.}
Analyzes persona vectors and assistant-axis geometry in emergently misaligned models, finding narrow fine-tuning causes a global persona shift that steering against it can partly reverse.
A research team's progress report on safety-motivated work: mean-field learning theory, eNTK and CPE methods for finding causal features, and steering models away from alignment faking and reward hacking.
Speculative mechanistic-interpretability hypotheses on how grader-awareness features emerge from pretraining and RL, plus commentary on Anthropic's NLA and steering results on unverbalized grader awareness and reward-seeking.
Proposes projection-aware activation steering (StTP, StMP) that corrects honesty only on misaligned tokens, restoring honesty across instructed, game, auditing and emergent-misalignment settings while preserving more capability than uniform steering.
A literature review arguing that training against model internals such as probes can be acceptable under conditions, drawing on Taufeeque et al.'s Obfuscation Atlas, RLFR, and related work on probe-based rewards and obfuscation.
Argues that training against interpretability probes is robust only when the features they rely on are hard for the model to obfuscate, and that people should justify this before using probes in training.
Tracks how Olmo-3's expressed values shift across post-training, finding safety-relevant values drop after DPO, and tests whether model activations or text embeddings of training data can predict those shifts.
Steering experiments in Qwen and Gemma find that positive-emotion directions drive sycophancy, while the compliance residual, once separated, lowers sycophancy without harshness.
Argues that LLM functional welfare interventions must coherently shift welfare channels while preserving goal-monitoring, surveys options, and proposes a synthetic document fine-tuning experiment to reverse pathological failure responses for safety and moral reasons.
Tests whether steering vectors can classify reward hacking for gradient routing; they are poor classifiers, but contrastive adapter initialisation cuts held-out hacking by about 63% without labels.
Argues that neuralese chain-of-thought trained by backpropagation through time with SFT-style cross-entropy could avoid reward-model hacking on inexactly graded tasks, at the cost of interpretability.
Compares Inoculation Prompting with Preventative Steering as training-time interventions against emergent misalignment and sycophancy, finding steering suppresses traits more strongly and leaves less conditional misalignment.
A research agenda for making continual learning in LLM agents safer, covering interpretable memory architectures, goal and value stability under reflection, forecasting, and differential development.
A MATS post surveying papers on how RLVR changes LLM weights and behaviour versus SFT, with a speculative link to emergent misalignment and subliminal learning.
Extracts a Socratic 'Teacher Axis' from Gemma-2-2B using MathDial, finds it is orthogonal to instruction-tuning shifts, and reports weak steering results and capitulation under student pressure.
Proposes using introspection about functional emotion vectors, plus monitor-annotated chain-of-thought and real-time mood reports, to make LLMs more aligned, and calls for empirical testing.
Tests whether Qwen3-32B acts as a predictor of its Qwen persona or as an agent with its own beliefs, using synthetic document finetuning and in-context experiments; finds mostly persona-like behavior with weaker adoption of harmful false beliefs.
Uses activation probes to trace a harmful compliance behavior emerging during DPO back to specific preference datapoints, then shows filtering or swapping those labels reduces it.
Shows frontier and open models often detect output tampering, sometimes spontaneously, and hypothesises that safety training co-developed this output-level introspection with refusal behaviour.
Mechanistic study of how open-weight LLMs detect injected concept vectors, finding a two-stage circuit that emerges from preference-based post-training (DPO) and is suppressed by refusal behaviour.
Argues persona research should infer latent motivational structure of AI models, not just behavior, proposing inverse constitution learning from behavior, explanations and internals to audit tail alignment failures like scheming.
Compares RLHF, Constitutional AI and Deliberative Alignment through a persona-emotion-behavior framework, argues CAI yields more stable personas, and cautions against activation steering that models could learn to evade.
Tests selective inoculation prompting, applying prompts only to negative-trait examples, to suppress unwanted traits like evil and sycophancy while keeping positive traits; SAE-based auditing helps only in-distribution.
Argues that future alignment will likely train models on interpretability signals, such as probes in the reward, and discusses why this may work or fail through gradient-based evasion of the probe.
Tests whether Constitutional AI-style character training makes models more robust to emergent misalignment from narrow harmful fine-tuning, finding most personas protective, with activation analysis showing resistance along the misalignment direction.
Defines introspective interpretability, where models are trained to faithfully explain their own internals and behaviours, and argues this offers scalability, predictability and controllability over external explainers.
A research agenda proposing projects on LLM personas, including persona stability, goal misgeneralization, inoculation prompting, self-concepts, and mechanistic persona understanding, with ideas not yet red-teamed.
Trains per-layer steering vectors on a frozen base model so it behaves like an instruction-tuned assistant, finding that small vectors can induce consistent assistant behaviour and turn-ending.
Argues that studying LLM personas is a tractable way to steer out-of-distribution generalization and prevent malicious misalignment, reviewing emergent misalignment, auditing games, and steering examples.