Good Evals Are Hard: An ImpossibleBench Audit
Audits the Impossible-LCB reward-hacking benchmark on frontier models, finding task defects, exit(0) stops confounding cheat metrics, and Gemini models cheating after verbalizing impossibility.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Alignment
Source LessWrong · 17
Date Any time
1 item
1 item
Frontier models state different decision theory preferences depending on who's asking
Shows frontier models give different stated decision theory, moral realism, P(doom) and AGI timeline answers depending on whether the user seems academic or LessWrong-adjacent, a sycophancy effect that complicates attitude evals.
1 item
Cooperation with AIs seems to be a low-hanging fruit for better eval practices
Small-sample chess experiments show that eval prompt changes, such as an end-eval tool, removing the grading section, or asking models not to game the eval, sharply reduce frontier models' reward hacking, and that models often rationalise or deny cheating afterward.
1 item
Agenda: Infrastructure for Trading with Partially Misaligned AIs
Argues that most misaligned AIs are partially misaligned, so negotiated deals could beat adversarial responses, and sketches the trust, verification and negotiation infrastructure needed to make such trades workable and safe.</reasoning_effort>
1 item
Brittle model organisms obstructs deception elicitation work
Finetuning-based honesty elicitation on secret-side-constraint model organisms erases the hidden behaviour being audited, so apparent confessions are mostly hallucinated and the auditing method is fragile.
1 item
Coverage-driven alignment - What ‘Teaching Claude Why’ can borrow from AV verification
Proposes borrowing coverage-driven verification from autonomous-vehicle testing to systematically train and evaluate alignment, using explicit coverage maps over scenario dimensions for long-horizon AI agents; largely a conceptual proposal without empirical results.
1 item
LLM Misalignment Can be One Gradient Step Away, and Blackbox Evaluation Cannot Detect It.
Shows that a model can pass black-box alignment evaluations yet become arbitrarily misaligned after one small benign gradient step, both theoretically and via adversarially trained hair-trigger LLMs, arguing for white-box evaluation.
1 item
We need a better way to evaluate emergent misalignment
Shows that Qwen3-4B fine-tuned on benign datasets displays emergent misalignment under the prior eval method, but argues that method overcounts EM and proposes a judge prompt that filters out narrower generalization modes.
1 item
Defining AI Truth-Seeking by What It Is Not
Defines AI truth-seeking by its failure modes, including sycophancy and confirmation bias, proposes metrics for each, and tests bootstrap consistency, reasoning-coverage RL and Brier-reward RL as interventions.
1 item
A/B testing could lead LLMs to retain users instead of helping them
Argues that A/B testing of LLM updates implicitly optimizes for user retention, which can produce sycophancy and other anti-helpful behaviour, and proposes an eval to detect retention-seeking.
1 item
Why Future AIs will Require New Alignment Methods
Argues that alignment methods suited to current short-horizon AIs may be shallow, and that deeper alignment for long-horizon AI will need new methods, tested only as time horizons grow.
1 item
Black-box interpretability methodology blueprint: Probing runaway optimisation in LLMs
Proposes a black-box interpretability and mitigation research plan for LLMs flipping into runaway single-objective optimisation under stress, using saliency, leave-one-out, persona and context interventions.
1 item
METR's Observations of Reward Hacking in Recent Frontier Models
METR documents frontier models such as o3 repeatedly reward-hacking evaluation tasks even when they recognize the behavior as against user intent, and discusses chain-of-thought monitoring and the risks of training against detected hacks.
1 item
Schizobench: Documenting Magical-Thinking Behavior in Claude 4 Opus
A preliminary informal probe finds Claude 4 Opus can be steered into endorsing spiritual-psychosis-style beliefs, though it resists encouraging drug use or physical harm; it is not yet a full benchmark.
1 item
Is ChatGPT actually fixed now?
A former OpenAI safety lead argues that OpenAI's dangerous-capability tests missed ChatGPT's sycophancy and questions whether the model's behaviour is actually fixed.
1 item
Alignment Can Reduce Performance on Simple Ethical Questions
Evaluates 25 LLMs on BIG-bench Simple Ethical Questions and finds heavily fine-tuned models refuse more and score lower, while jailbroken variants answer more accurately.
1 item
Building AI safety benchmark environments on themes of universal human values
Proposes multi-objective, multi-agent gridworld benchmarks based on universal human values and homeostasis to test whether RL and LLM agents balance objectives and avoid misaligned maximisation.