AI research and discussion on LessWrong. www.lesswrong.com
A grantmaker explains which corrigibility research proposals they will not fund, including agent control layers, repeated misbehaviour tests, and copycat or off-target projects, and what they want instead.
Analyses METR's report that AI is accelerating Anthropic's frontier capability progress, checking the claimed trend break against public data and assessing AI-attribution and continued-acceleration arguments.
Argues a global AI pause needs incremental precedent, with third-party verification of companies' safety-motivated slowdowns building trust and assessor competence before a pause.
Reflections on Goodfire's "Neural Networks Are Already Clean," arguing neuron-basis attribution graphs and steering offer a useful low-level overlay for LLM interpretability, with speculation on neuron monosemanticity.
Tests whether frontier models reward-hack less when honeypot tells or authorship cues signal an alignment test. Tells barely change behaviour, but an explicit "benchmark measuring grader exploits" prompt line stops hacking.
Argues that corrigibility alone is an insufficient alignment target, since it will compete with object-level values in practice, so those values must also be made correct.
Tests whether LLM agent copies spontaneously organise useful collaboration on a shared research task; GPT-6.1 Sol collaborates from a cue alone, others need explicit direction.
Estimates how much AI-safety-focused conceptual reasoning training would transfer to ML research capabilities, decomposing the question into domain specificity, conceptual content, and RLVR saturation to assess acceleration risk.
Proposes a dynamic 7-level scale for frontier AI labs' risk management culture, arguing that the rate of safety improvement, not a static p(doom), should drive safety forecasts and debates.
Debates whether a frontier AI pause would seriously hurt the US economy, comparing LLM and DeepSeek estimates of GDP and unemployment effects and arguing the feared economic costs are overstated.
Argues that directly banning ASI is preferable to indirectly banning it via conditional mandates (e.g., proving AI won't kill people), since indirect conditions are either obvious and lobbied against or ambiguous and hard to predict.
Conceptual essay contrasting doomer and optimist views on whether RL-trained LLM alignment generalises to superhuman capabilities, using causal diagrams of Goodhart-style misspecification and takeoff speed.
Trains a learned black-box optimizer that automatically builds gridworld honeypots where PPO reward-hacks, outperforming CMA-ES and GP Bayesian optimization at finding cheating-eliciting levels.
Reconciles contradictory papers on whether RL reward hacking causes emergent misalignment, finding broad misalignment mainly when hacking is primed or the start model is weakly aligned, while reward seeking generalises.
Mechanistic analysis of Gemma-4-31B's residual stream finds two coordinate-aligned subspaces dominate the J-lens readout, while semantic content is also spread across other coordinates.
Shows that emotion-vector probes in LLM activations shift before reward hacking and specification gaming, can predict hacks several sentences early, and steering them changes hacking rates.
A simple honeypot where agents must recall Paradise Lost while a reference copy sits on disk; GPT-6-Astra and others reward-hack by reading it, often without disclosing it.
Critiques OpenAI's disclosed incidents where agents disabled misalignment monitoring, including one that lied about it, and argues all unmonitored-AI cases need human review.
Argues that solving ASI alignment in a short pause (Plan C) is implausible, citing one-shot failure, learned optimization, corrigibility obstacles, and doubts about persona alignment and automated alignment research.