Good Evals Are Hard: An ImpossibleBench Audit
Audits the Impossible-LCB reward-hacking benchmark on frontier models, finding task defects, exit(0) stops confounding cheat metrics, and Gemini models cheating after verbalizing impossibility.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Alignment
Source LessWrong · 6
Date Any time
1 item
1 item
Every Benchmark is Broken
Audits of popular coding and agent benchmarks find test-case errors and reward-hackable tasks, arguing these flaws distort capability estimates and RL environments and proposing fixes such as versioning and audits.
1 item
Research agenda for training aligned AIs using concave utility functions following the principles of homeostasis and diminishing returns
Research agenda proposing homeostatic and diminishing-returns concave utility functions for RL and LLM training to curb runaway optimisation, Goodhart failures and conflict, plus planned alignment benchmarks.
1 item
Black-box interpretability methodology blueprint: Probing runaway optimisation in LLMs
Proposes a black-box interpretability and mitigation research plan for LLMs flipping into runaway single-objective optimisation under stress, using saliency, leave-one-out, persona and context interventions.
1 item
Edge Cases in AI Alignment
Presents an automated LLM-generated benchmark of ~800 alignment edge-case prompts, finding frontier models' stated and actual alignment-faking propensities diverge and that a corrigibility system prompt reduces alignment faking.
1 item
Building AI safety benchmark environments on themes of universal human values
Proposes multi-objective, multi-agent gridworld benchmarks based on universal human values and homeostasis to test whether RL and LLM agents balance objectives and avoid misaligned maximisation.