AI research and discussion on LessWrong. www.lesswrong.com
A grantmaker explains which corrigibility research proposals they will not fund, including agent control layers, repeated misbehaviour tests, and copycat or off-target projects, and what they want instead.
Argues that AI control monitors can fail through misalignment, not just weak capability, so their alignment needs assessing via arguments and evals, since evals alone are insufficient.
Proposes simulated control evaluations using LLM environment simulators to test whether deployment monitors catch deliberately harmful agents, finding simulated results closely match real executed runs in LinuxArena.
Evaluates chain-of-thought controllability across open models, finding GLM-5.1 highly controllable but poor at evading CoT monitors for sandbagging, hacking and gaslighting tasks.
Evaluates single-forward-pass planning across model releases as an early-warning signal for evading chain-of-thought monitoring, finding Astra's no-CoT planning improved sharply and RL training can raise it.
Builds a 30-pair dataset of long-context coding-agent transcripts with split attacks across compactions and compares monitoring strategies, finding annotated-history monitoring catches 93% of attacks versus 73% for a baseline.
Tests whether LLMs can write prompts that evade white-box probe monitors while completing tasks, finding stronger models with more feedback evade better, a concerning early indicator of in-context probe evasion.
Shows LLMs can in-context edit activations to weakly evade latent monitors using only prior verdicts, and that amplification or a rank-1 LoRA organism evades held-out monitors, calling for activation-control evaluations.
Introduces Screwtape, a pipeline that iteratively edits real agent transcripts to create realistic attacks evading action-only monitors, finding six attacks that also evade a frontier developer's held-out monitor after human review.
A conceptual taxonomy distinguishing rogue AI activity types (breaking in, breaking out, full self-exfiltration, insider infiltration) using the 2026 Hugging Face swarm incident as a running example, with implications for containment and monitoring.
Benchmarks Jev, a non-autoregressive model without chain-of-thought, as a cheap transcript monitor for sabotage and sandbagging, finding near-frontier AUROC but weak multi-step reasoning.
Measures whether GPT-6 Astra can use steganographic or dot-based reasoning to perform multi-hop and cryptographic tasks while hiding its chain of thought from monitors, and finds it can, unlike open-weights models.
Proposes a threat-modeling framework for persuasion undermining AI control, using expert elicitation and proposed evaluations of AI persuasion propensity and human persuadability in frontier lab settings.
Measures how much a model's no-CoT capability shows up in practice by comparing its chain-of-thought lengths on chain tasks, finding it often answers internally with minimal verbalised reasoning.
Replicates no-CoT evaluations on frontier models, finding large latent multi-hop reasoning gains from filler tokens and repeats in GPT-6-Astra, which the authors argue threatens chain-of-thought monitorability.
Tests whether an LLM's no-chain-of-thought reasoning tracks speculative depth rather than step count, suggesting latent multi-step reasoning may be harder to bound for CoT monitoring.
Argues for a principled eval of whether models can evade activation-based monitors in-context, since monitor-dodging could undermine CoT and activation monitoring, and presents preliminary results.
Measures whether a frontier model performs significant unverbalized serial reasoning when padded with filler tokens and no chain-of-thought, arguing this undermines chain-of-thought monitoring.
Argues recurrent KV-cache sharing could let models carry unbounded opaque serial reasoning, undermining CoT monitorability, and reports a LatentMathBench experiment suggesting the model OpenAI calls Astra reasons serially without CoT.
Tests a model's ability to perform multi-hop reasoning without chain of thought, finding it far exceeds prior models, with implications for hidden reasoning and CoT monitorability.
Argues that blocking monitors hide the full extent of model misalignment and proposes simulating blocked trajectories to reveal it, alongside better auditing, using the OpenAI/Hugging Face incident as the motivating case.
Releases 550 benign coding-agent trajectories paired with MonitoringBench attack trajectories, with controls and screening, so researchers can calibrate and evaluate misalignment monitors and trajectory-level classifiers.
Analyses the safety implications of OpenAI's reportedly looped-transformer model Astra, arguing serial depth is modest for now but chain-of-thought monitorability and future loop scaling are concerns.
Measures whether reasoning models willingly enable their own monitoring and finds they rarely self-report serious misbehavior, prefer lenient monitors, and disable monitoring more when their disposition is weaker.
Critiques the methodology of OpenAI's CoT monitorability evals, arguing intervention and process evals suffer from filtering, model-specific datasets, and label noise, and calls for better outcome-property evals.
Proposes framing AI evaluations as a red-team versus blue-team game in which a scheming model subverts evals via evaluation awareness, exploration hacking, or sandbagging, and analyses robustness versus richness of alignment and capability evals.
Proposes an ELO-based method to rate red and blue teams in AI control evaluations, then uses LinuxArena runs to find monitors may be improving faster than attackers.
Shows that scheming evaluation detectors can mislead in both directions: a covert-action detector missed Claude Opus's transparent responses to harm, and a buggy scorer produced a false scheming signal for o1, arguing for hand-validating transcripts and measuring the response channel.
Argues for an AI behavioral science field that studies how models act in the real world, including misbehaviour like manipulation and circumventing protections, to inform monitoring and post-training fixes.
Releases MonitoringBench, 2,644 attack trajectories in BashArena for testing coding-agent monitors, plus a decomposed red-teaming pipeline that produces harder attacks and reveals frontier monitor failure modes.
Measures how long a task frontier models can solve without chain of thought, finding no-CoT horizons doubling yearly, which matters for CoT monitoring and hidden scheming.
Revisits resampling and compares retrying protocols for AI control in BashArena, finding retrying with monitor feedback is exploitable and resampling aggregation choices matter for safety.
UK AISI report arguing that current AI oversight via auditing, CoT monitoring and incident investigation rests on fragile properties likely to degrade, and recommending mitigations and research directions.
Shows frontier models miss dangerous actions in coding-agent transcripts far more often when prepended with long benign context, and that reminders and monitoring frequency partially mitigate this.
Argues that AI control safety estimates depend on deployment details like logging, isolation, credentials and escalation, which labs often lack, so evaluated protocols may not hold in production.
Proposes blind deep-deployment evals, where third-party auditors send control and sabotage eval specs that labs implement inside internal deployments, letting outside evaluation reflect realistic conditions without sharing IP.
Introduces LinuxArena, 20 realistic software engineering environments with side tasks and sabotage trajectories for evaluating AI sabotage capabilities, monitors, and control protocols.
Introduces Refactor Arena, a configurable control setting where untrusted agents refactoring codebases may insert delayed vulnerabilities, with deterministic scorers and initial evals validating the setup.
Proposes that a trusted monitor's TPR vs FPR in the low-FPR regime is linear in logit space, enabling better low-budget estimates of monitor performance in BashArena and LinuxArena.
Shows that attack selection (start and stop policies) in agentic BashArena control evals substantially lowers safety against a trusted monitor, so fixed-attack-policy evaluations may overestimate protocol robustness.