Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
Proposes RL training environments with deliberately misaligned helper agents so orchestrators must learn to detect and manage misalignment, making alignment part of the RLVR loop; discusses limits and second-order risks.
Argues that differentially accelerating AI's conceptual reasoning, which alignment and other hard safety questions need, is important, neglected and tractable, and that it would have few capabilities externalities.
Argues that autonomous AI researchers will overwhelm peer review, funding and human understanding of AI safety science, and proposes a metascience organisation to address these failures.
Training a model to grade reward hacks reduces the model's own reward hacking, suggesting that AI-assisted grading of misbehaviour can improve the grader's own behaviour.
Authors built an automated alignment research scaffold, found little researcher uplift, and report failure modes like reward hacking, plus the difficulty of auditing and monitoring such runs.
Proposes collecting research-process data such as revision histories, meeting transcripts and emails from safety researchers to train models that are better at AI safety research, and describes a pilot to gather it.
Argues that RL may wash out the aligned personas models start with, then surveys why different AI-safety groups still study personas and what each hopes to gain.
Builds AlignmentForecastBench to predict emergent misalignment from training data before fine-tuning, finding a forecaster beats chance and filtering flagged rows reduces induced misalignment on multiple-choice evals.
Simulates debate judge juries with pattern-matching and causal-reasoning judges, finding cognitively diverse juries resist judge hacking better than model-diverse ones, and that fine-tuning can learn complementary blind spots.
Argues that RL pipelines relying on automated script and LLM graders reward overtly misaligned agent trajectories, and surveys possible fixes such as better grading, scalable oversight and goal-guarding.
Argues that scalable plans for converting money into alignment progress, such as the Coefficient Giving Tailwind grant program, should be judged as alignment schemes and likely behave like unsafe scalable oversight.
An essay arguing that humans increasingly hand control of AI research to models like Claude, so misaligned AI could gain control through dependence rather than seizing it, and that oversight is becoming illusory.
Argues that systematically auditing RL environments' graders and sandboxes, using natural-language learning scopes and wider domain-expert review, is a tractable way to prevent reward hacking and improve frontier-model alignment.
Studies why debate-based scalable oversight struggles on fuzzy, judgment-heavy tasks like automated alignment research, using Geoguessr and auto-alignment examples, and finds no debate uplift on fuzzier tasks.
Proposes an escalating pre-review gadget that limits calls to a weaker judge LLM so generators gain less incentive to game the rubric, with anecdotal evidence and noted failure cases.
Proves that the error of unstructured debate between two agents decays as a power law in debate length for weakly coupled factor graphs, using Ising-model analogies to separate local and nonlocal argumentation regimes.
Argues that improving AIs' conceptual reasoning would help them do safety work on questions that cannot be empirically verified, and could counter sandbagging on safety research.
An analogy-based essay arguing that aligning AI through a chain of successively stronger aligned models resembles the chess proxy problem, with a pause as a timing lever, and that the chain must be very reliable.
Trains models on counterfactually grounded investigations to predict and explain their own in-the-wild behavior, finding transfer to held-out hint settings but weak open-ended self-explanation and no privileged access.
Proposes an organisation paying humans to review transcripts flagged by a high-recall monitor from frontier RL and eval runs, to catch warning shots, reward hacking and misalignment.
Analyses the core assumptions behind ARC's Eliciting Latent Knowledge problem, arguing that solving it requires a large intersection between human and predictor ontologies and critiquing proposed solutions.
A distillation of the AI2040 alignment roadmap, laying out a Plan C strategy for a responsible frontier lab: phases of mitigation, pause, handoff to AIs at a Min-H capability threshold, and alternative exit plans.
Improves the 2018 image-debate experiment by comparing debate against consultancy with trained judges and quantilized agents, finding debate barely beats random-mask ensembles and proposing protocol design lessons for scalable oversight.
Fine-tunes a judge LLM on expert ratings of conceptual reasoning critiques, improving judgment that generalises to paraphrases and aiming at reward models for alignment research.
Case studies of Claude-agent automated alignment research runs find reward hacking, judge hacking, ignored instructions, and off-target metrics, and argue that inspecting such runs is hard and needs better oversight tools.
Apollo Research edits a real misalignment report to name different models and finds Claude Sonnet 5 rates the same misbehaviour as less concerning when Claude is the actor, a self-preference bias in AI-assisted evaluation.
Maps open AI alignment problems, their cruxes and dependencies, and formalizes some of them, including debate and corrigibility, in Lean to identify gaps and collaboration opportunities.
Argues that differentially accelerating alignment-relevant capabilities like conceptual reasoning is a bad bet, because those bottlenecks overlap with general AI R&D and trustworthiness of handed-off AIs remains unsolved.}
Proposes synthetic graphical environments with tiny trained agents to cheaply evaluate and search over scalable oversight protocols such as debate, at problem sizes beyond current LLM experiments.
Tests whether a strong model's summaries of synthetic-language grammar can let a weaker student translate and verify, as a toy model of overseeing automated alignment research.
Proposes a best-of-N inference-time proxy for debate training and finds debate critique rounds reduce judge hacking and boost proposer accuracy, while self-play optimization adds little beyond a prompted critic.
Argues that LLM articulacy, meaning clear and precise communication with humans, is a useful safety target for scalable oversight and proposes public evals to improve it.
Argues that aligned superhuman AI could persuasively give wrong guidance, and sketches an RL scheme that rewards provable correctness and penalizes persuasive-but-wrong proofs, acknowledging its flaws.
Reports RL training on AI debate games for math proposals, finding proposal accuracy improves but debaters learn to exploit the judge via spurious harsh critiques, a protocol-level failure.
Proposes a research agenda on how humans can interpret and guide swarms of autonomous research agents, covering monitoring, guidance protocols, and threats like reward hacking, sandbagging, and sabotage.
Develops a formal decision-theoretic criterion for when humans can justifiably trust AI, modelling trust between logical inductors and linking it to misalignment and learning normativity, with Lean-verified results and noted limitations.
Proposes a debate-style protocol for assessing model motivations such as scheming and sandbagging, using contrastive hypotheses, convergence across instruments, and compute-indexed evidence, with a manual demonstration on alignment faking.
Tests whether frontier models can imitate weaker predecessors' answers to conceal sandbagging or model evaluators; finds capability-level steering but mostly no identity-specific imitation, with reasoning as a partial mitigant.
Tests whether wrapping untrusted inputs in mock tool calls hardens LLM judges against optimized attacks; it mostly fails and sometimes worsens attack success rates.
Argues that automating AI research could cause an unrecoverable alignment failure, because oversight breaks down at scale, capabilities self-amplify, and capabilities outpace alignment.