AI research and discussion on LessWrong. www.lesswrong.com
Builds AlignmentForecastBench to predict emergent misalignment from training data before fine-tuning, finding a forecaster beats chance and filtering flagged rows reduces induced misalignment on multiple-choice evals.
Studies why debate-based scalable oversight struggles on fuzzy, judgment-heavy tasks like automated alignment research, using Geoguessr and auto-alignment examples, and finds no debate uplift on fuzzier tasks.
Improves the 2018 image-debate experiment by comparing debate against consultancy with trained judges and quantilized agents, finding debate barely beats random-mask ensembles and proposing protocol design lessons for scalable oversight.
Apollo Research edits a real misalignment report to name different models and finds Claude Sonnet 5 rates the same misbehaviour as less concerning when Claude is the actor, a self-preference bias in AI-assisted evaluation.
Argues that differentially accelerating alignment-relevant capabilities like conceptual reasoning is a bad bet, because those bottlenecks overlap with general AI R&D and trustworthiness of handed-off AIs remains unsolved.}
Proposes synthetic graphical environments with tiny trained agents to cheaply evaluate and search over scalable oversight protocols such as debate, at problem sizes beyond current LLM experiments.
Argues that LLM articulacy, meaning clear and precise communication with humans, is a useful safety target for scalable oversight and proposes public evals to improve it.
Introduces a v0.1 dataset of expert-rated critiques of arguments and LLM prompts for judging and writing critiques, aiming to make AI assistants useful for conceptual AI safety research sooner.
Introduces Petri, an open-source framework where AI agents automatically audit target models across multi-turn scenarios, surfacing deception, oversight subversion and other misaligned behaviours across 14 frontier models.
Argues that agent evaluations are hard to build and that overseeing reward-hacking agents requires scalable monitoring, interfaces, and auditing agents to avoid being misled by proxy results.
A video explaining scalable oversight, debate, and sandwiching, and how Bowman et al. ran a basic sandwiching experiment with non-experts overseeing a language model.
Empirically measures weak-to-strong oversight across four games using LLM Elo curves, then theoretically models nested scalable oversight to estimate how often weaker overseers can control stronger adversaries.