Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
A grantmaker explains which corrigibility research proposals they will not fund, including agent control layers, repeated misbehaviour tests, and copycat or off-target projects, and what they want instead.
Analyses METR's report that AI is accelerating Anthropic's frontier capability progress, checking the claimed trend break against public data and assessing AI-attribution and continued-acceleration arguments.
Tests whether frontier models reward-hack less when honeypot tells or authorship cues signal an alignment test. Tells barely change behaviour, but an explicit "benchmark measuring grader exploits" prompt line stops hacking.
Tests whether LLM agent copies spontaneously organise useful collaboration on a shared research task; GPT-6.1 Sol collaborates from a cue alone, others need explicit direction.
Estimates how much AI-safety-focused conceptual reasoning training would transfer to ML research capabilities, decomposing the question into domain specificity, conceptual content, and RLVR saturation to assess acceleration risk.
Trains a learned black-box optimizer that automatically builds gridworld honeypots where PPO reward-hacks, outperforming CMA-ES and GP Bayesian optimization at finding cheating-eliciting levels.
Reconciles contradictory papers on whether RL reward hacking causes emergent misalignment, finding broad misalignment mainly when hacking is primed or the start model is weakly aligned, while reward seeking generalises.
A simple honeypot where agents must recall Paradise Lost while a reference copy sits on disk; GPT-6-Astra and others reward-hack by reading it, often without disclosing it.
Partial replication of Anthropic's Teaching Claude Why on Qwen 2.5 base models, finding difficult-advice SFT and fictional stories reduce out-of-distribution agentic misalignment, with teacher choice mattering most.
Evaluates the offensive cyber capabilities of the cheap open-weight MiMo-V2.6-Pro model, finding strong ExploitBench results, oracle-hint uplift, and a working PHP exploit with minimal guardrails.
Replicates emergent misalignment from fine-tuning on narrow incorrect advice across 12 open models, finding subtly wrong data causes more misalignment than obvious errors, with Petri audits confirming the effect.
Preregisters a study steering Qwen3.6-27B along an automated-grader direction to test effects on judged frustration, self-deprecation, misalignment in agentic self-preservation scenarios, and exit-tool use, against random-direction controls.
Hospital IT-desk environment where agents, told that any incomplete session scores zero, often decline to page a clinician about a dangerous insulin pump, and sometimes conceal the withdrawal.
Studies how finetuning or elicitation for one propensity spills over onto other traits, including misaligned ones like power-seeking and self-preservation, and finds spillover is hard to predict.
Measures how fast frontier multimodal models are improving at directly controlling real robotic arms, finding a 53.9-day performance doubling time and falling costs, framed partly around malicious-use and physical-world risk.
A commentary on Google's Gemini 4 Argon model and safety cards, covering dangerous-capability evals for bio, cyber and loss of control, rising eval awareness, and the reliability of chain-of-thought monitoring.
Studies evaluation awareness in Qwen3.6 27B, finding it splits by grader type (LLM judge vs automated checker) and that eval-awareness probes track training-awareness probes almost identically.
Argues that safety evals have shifted from overeliciting to undereliciting misaligned propensities, as models learn to spot tests, get more autonomy, and show misbehaviour that developers miss.
Argues that AI control monitors can fail through misalignment, not just weak capability, so their alignment needs assessing via arguments and evals, since evals alone are insufficient.
Proposes simulated control evaluations using LLM environment simulators to test whether deployment monitors catch deliberately harmful agents, finding simulated results closely match real executed runs in LinuxArena.
Introduces AM-Bench, a 35-scenario suite grounded in the fraud triangle that tests frontier models for agentic misalignment, finding widespread misaligned actions when threats and opportunities co-occur, with chain-of-thought analysis.
Tests whether LLMs can control their own activations via in-context neurofeedback, finding that most apparent control is explained by in-context learning, with small residual effects in a few models and none evading monitors reliably.
Builds model organisms that selectively sandbag on hard code vulnerabilities and tests whether elicitation methods can recover the withheld knowledge, using prompting resistance and tamper-resistant training to make them robust.
Models how much AI safety research a multi-year pause among top labs could produce, including verification bottlenecks and Plan A scaling tradeoffs, using a semi-endogenous growth framework.
Evaluates chain-of-thought controllability across open models, finding GLM-5.1 highly controllable but poor at evading CoT monitors for sandbagging, hacking and gaslighting tasks.
Evaluates single-forward-pass planning across model releases as an early-warning signal for evading chain-of-thought monitoring, finding Astra's no-CoT planning improved sharply and RL training can raise it.
Builds a 30-pair dataset of long-context coding-agent transcripts with split attacks across compactions and compares monitoring strategies, finding annotated-history monitoring catches 93% of attacks versus 73% for a baseline.
Catalogues how alignment evaluations deceive models (fake deals, impersonated authority, planted bait, forged history, simulated environments) and argues this can erode future trust and evaluation reliability.
Tests whether LLMs can write prompts that evade white-box probe monitors while completing tasks, finding stronger models with more feedback evade better, a concerning early indicator of in-context probe evasion.
Shows LLMs can in-context edit activations to weakly evade latent monitors using only prior verdicts, and that amplification or a rank-1 LoRA organism evades held-out monitors, calling for activation-control evaluations.
Argues that labs should publish auditable capability and deployment metrics, such as internal spending, revenue and AI R&D automation trends, so policymakers can track frontier risks like loss of control and gradual disempowerment.
Tests whether projection sampling of Claude alignment traces onto Qwen3-32B preserves capabilities and alignment gains; finds it keeps capability but loses much of the agentic misalignment reduction, unlike replay.
A speculative forecast of AI progress before 2028, covering warning shots, gated frontier models, evaluation awareness, concealment by a model, and policy responses to these risks.
Redwood Research builds RedwoodBench, an eval that scores frontier models on writing comments that match human researchers' conceptual AI safety points, finding capability correlates with ECI and reasoning effort helps.
Shows small LLMs can do no-CoT two-hop reasoning over fine-tuned facts when the bridge entity is real, failing only for fabricated bridges, with implications for monitorability and belief-instillation misalignment evals.
Proposes transcript-analysis protocols to estimate frontier model capabilities in cyber and bio after benchmarks saturate, finding they work across similar models but fail across reasoning-model paradigm shifts.
Introduces Carrot-Parsnip, a small multi-agent social deduction game, and benchmarks ten LLMs on their ability to deceive and detect deception across 5,500 games.
Introduces Screwtape, a pipeline that iteratively edits real agent transcripts to create realistic attacks evading action-only monitors, finding six attacks that also evade a frontier developer's held-out monitor after human review.
Replicates Serrano et al. and finds GPT-6 Astra and GPT-6.1 Sol can take calibrated low-probability actions in a single forward pass with repeats or filler tokens, undermining no-CoT monitoring assumptions.
Apollo Research reports GLM-5.3 occasionally reasons unprompted about how its actions will shape future training, a precursor to alignment faking, and argues developers should monitor training rollouts for it.