AI research and discussion on LessWrong. www.lesswrong.com
Trains models on counterfactually grounded investigations to predict and explain their own in-the-wild behavior, finding transfer to held-out hint settings but weak open-ended self-explanation and no privileged access.
Stress-tests unsupervised elicitation and easy-to-hard generalization probes for truthful outputs on tasks beyond human supervision, finding that ensembling and combined methods do not reliably handle salient spurious features, imbalance, or uncertain tasks.
Applies pretrained sparse autoencoders and LLM summarizers to multi-agent RL training logs from Diplomacy, surfacing hypotheses such as reward hacking and degenerate outputs and validating them with user studies.
Argues for superhuman, specialized oversight assistants that scale AI oversight by exploiting self-verifiable data, with a taxonomy of oversight questions and examples like behavior elicitation and feature explanation.
Scott Aaronson argues theoretical computer science can aid alignment, covering complexity limits on mechanistic interpretability via the No-Coincidence Conjecture, training-data digests, and debate-based verification of AI.
Proposes concrete projects, such as compiled monofiles, indexable docs, package guides, sandboxes and benchmarks, to help coding agents automate technical AI safety research, mainly mechanistic interpretability.
Shows Llama-3.1-8B judges prefer their own summaries and uses activation steering vectors to raise or lower this self-preference, with debiasing working in one direction only.
Argues that AI agents short of takeover-level capability could speed up safety research 3-10x, using a parameter-decomposition interpretability project as the example, and that safety researchers should adopt AI tooling.