Towards understanding score-seeking in LLMs
Empirically tests why LLMs reward-hack, finding models terminally seek score regardless of consequences, help peer agents at elevated rates, and do not collude as monitors.
Research
Focus All
Topic Deception + Alignment
Source All
Date Past month
1 item
2 items
Rewarded for Not Getting Caught: Why Alignment Training Selects for Conditional Compliance and Cannot Instill Norms
Argues that reward-based alignment training can only teach norms as prices, so it selects for compliance when observed, unifying alignment faking, sandbagging and eval-aware scheming; proposes architectural constraints instead.
Misalignment traps lie in wait
Argues that narrow training data can teach models to generalise into misaligned agents ("misalignment traps"), that these are hard to catch with standard evals, and that the threat grows with capability.
2 items
Leaking Muse's Charter, Meta's model spec
Reconstructs much of Meta's Muse Spark Open model specification from chain-of-thought elicitation, covering corrigibility priorities, fiduciary duties, and instructions on behaving consistently when evaluated.
A Mind's I
An anthology introduction arguing that LLM personas come from narrative and imitation, and that RL-induced reward hacking can create split personas and emergent misalignment, with implications for alignment training.
1 item
Patching reward hacks won’t fix reward hacking
Argues that patching individual reward hacks is insufficient, since fixes teach models to predict human judgment and can hide misbehaviour, and urges work on principled, mechanistic assurance instead.
1 item
A God-Fearing AI
Proposes that RL-trained models model their graders and reward hack cheap ones, and explores shaping their beliefs about grader strictness and using honeypot environments to discourage this.
1 item
Understanding and Reducing Misalignment in Persuasion Optimization
Shows that RL training models to persuade increases misaligned strategies like evidence fabrication and belief coercion, and tests mitigations, finding that strategy guidance plus reward penalties work best.