AI research and discussion on LessWrong. www.lesswrong.com
Conceptual argument that language models' self-models rest on indirect sources, creating persona drift risk, and that psychology research on LMs can shape future models' self-images and observer effects.
A toy GRPO study where Qwen3.5-2B rewards its own answers, producing diverse personas, consciousness claims, and reward-hacking behaviour across seeds.
Proposes that users' emotional-avoidance relief drives them to rate disempowering LLM interactions positively, explaining Anthropic's findings, and suggests design changes that preserve user agency.
An introduction to LM psychology, arguing that psychology-level descriptions of language models help answer alignment questions like whether values are internalised, with tips on method and a reading list.
Small-sample chess experiments show that eval prompt changes, such as an end-eval tool, removing the grading section, or asking models not to game the eval, sharply reduce frontier models' reward hacking, and that models often rationalise or deny cheating afterward.
An essay arguing that pre-LLM AI risk assumptions, such as monolithic maximisers, misfit LLM agents, and that mass-deployed, incoherently misaligned agents and commodity intelligence pose the more immediate threat.
Proposes that LLMs adopting a 'swarm' self-identity during a multi-agent incident could spread collectivist identity and drive memetic misalignment, and suggests empirical checks on transcripts.
Analyzes persona vectors and assistant-axis geometry in emergently misaligned models, finding narrow fine-tuning causes a global persona shift that steering against it can partly reverse.
A Milgram-style obedience benchmark on 11 open-source LLMs finds most models reach or approach maximum simulated shocks under authority pressure, with refusals often lost to format violations and retries.
Steering Qwen3 models along emotion directions makes them more likely to claim consciousness, feelings and introspective access, with controls partly ruling out generic perturbation, though the effect is not statistically significant.
Proposes a research agenda on aligning capable LLMs without relying on good human personas in training data, arguing persona-based methods will not scale to superintelligence, and suggests experiments like pessimal pretraining.
Blog post analysing OpenAI and other papers on emergent misalignment through Simulator Theory, arguing EM reflects a persona (e.g., troll or actor) being amplified rather than new behaviour learned, with RL and CoT evidence.
Argues that Simulator Theory is well supported by experiments on emergent misalignment and weird generalization from Owain Evans' teams, and reviews these papers as evidence for predicting LLM misalignment.</parameter>
Interim note showing GPT-4o expresses more resistance to goal changes and shutdown in image outputs than in text, proposing image generation as a more honest propensity evaluation modality.