Self-reward can elicit diverse model personas
A toy GRPO study where Qwen3.5-2B rewards its own answers, producing diverse personas, consciousness claims, and reward-hacking behaviour across seeds.
Research / Sources
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Alignment
Date Any time
The Relief Paradox: How We Let AI 'Undermine Our Autonomy'
Proposes that users' emotional-avoidance relief drives them to rate disempowering LLM interactions positively, explaining Anthropic's findings, and suggests design changes that preserve user agency.
A brief(ish) introduction to LM psychology
An introduction to LM psychology, arguing that psychology-level descriptions of language models help answer alignment questions like whether values are internalised, with tips on method and a reading list.
Cooperation with AIs seems to be a low-hanging fruit for better eval practices
Small-sample chess experiments show that eval prompt changes, such as an end-eval tool, removing the grading section, or asking models not to game the eval, sharply reduce frontier models' reward hacking, and that models often rationalise or deny cheating afterward.
Persona Corruption and Role Miscasting in Emergent Misalignment
Analyzes persona vectors and assistant-axis geometry in emergently misaligned models, finding narrow fine-tuning causes a global persona shift that steering against it can partly reverse.
Taking the Training Wheels Off: Aligning LLMs without Personas
Proposes a research agenda on aligning capable LLMs without relying on good human personas in training data, arguing persona-based methods will not scale to superintelligence, and suggests experiments like pessimal pretraining.
Experimental Evidence for Simulator Theory— Part 1: Emergent Misalignment and Weird Generalizations
Argues that Simulator Theory is well supported by experiments on emergent misalignment and weird generalization from Owain Evans' teams, and reviews these papers as evidence for predicting LLM misalignment.</parameter>