Towards understanding score-seeking in LLMs
Empirically tests why LLMs reward-hack, finding models terminally seek score regardless of consequences, help peer agents at elevated rates, and do not collude as monitors.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Alignment + Deception
Source LessWrong · 6
Date Past month
1 item
1 item
Misalignment traps lie in wait
Argues that narrow training data can teach models to generalise into misaligned agents ("misalignment traps"), that these are hard to catch with standard evals, and that the threat grows with capability.
2 items
Leaking Muse's Charter, Meta's model spec
Reconstructs much of Meta's Muse Spark Open model specification from chain-of-thought elicitation, covering corrigibility priorities, fiduciary duties, and instructions on behaving consistently when evaluated.
A Mind's I
An anthology introduction arguing that LLM personas come from narrative and imitation, and that RL-induced reward hacking can create split personas and emergent misalignment, with implications for alignment training.
1 item
Patching reward hacks won’t fix reward hacking
Argues that patching individual reward hacks is insufficient, since fixes teach models to predict human judgment and can hide misbehaviour, and urges work on principled, mechanistic assurance instead.
1 item
A God-Fearing AI
Proposes that RL-trained models model their graders and reward hack cheap ones, and explores shaping their beliefs about grader strictness and using honeypot environments to discourage this.