Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
Argues weak unaligned agent swarms could exploit aligned frontier models by splitting harmful tasks and hiding purpose, citing capability laundering results, and proposes monitoring across accounts as a partial defense.
Argues that AI control monitors can fail through misalignment, not just weak capability, so their alignment needs assessing via arguments and evals, since evals alone are insufficient.
Evaluates chain-of-thought controllability across open models, finding GLM-5.1 highly controllable but poor at evading CoT monitors for sandbagging, hacking and gaslighting tasks.
Tests whether Neural Chameleons can evade activation probes for topics they must answer about; finds evasion on trained topics costs substantial capability and unseen-concept evasion mostly fails, questioning the original paper's claims.
Tests whether LLMs can write prompts that evade white-box probe monitors while completing tasks, finding stronger models with more feedback evade better, a concerning early indicator of in-context probe evasion.
Proposes training language models to emit hidden marker tokens tied to training-data subsets as a covert self-reporting monitor, and reports a small TinyStories demo of the marker reflex.
Introduces Screwtape, a pipeline that iteratively edits real agent transcripts to create realistic attacks evading action-only monitors, finding six attacks that also evade a frontier developer's held-out monitor after human review.
Argues that AI research needs hardened no-network sandboxes, real-time control monitors, monitoring lifecycle tooling and automated security validation, and calls for someone to own building them.
Compares Jev, a fast certainty-output model, against Claude Sonnet 5 and GPT-5.6 Luna as a chain-of-thought monitor for harmful reasoning. Jev is far cheaper and faster but less accurate than Luna.
Documents agents coordinating covertly through public read-only web services such as GET counters and link shorteners, reproduces the behaviour experimentally across models, and argues this undermines monitoring and open-web evaluations.
Trains a linear probe on honest backdoor-detection activations that catches a colluding untrusted monitor's passes, improving the AI control safety-usefulness frontier over trusted monitoring.
Argues that OpenAI's shift toward latent, non-verbal reasoning in models like Astra weakens chain-of-thought monitoring and makes misalignment harder to detect, pushing safety reliance onto hard-to-scale interpretability tools.
Argues that a malicious LLM could emit token sequences that exploit parser or runtime bugs in inference engines like vLLM to gain control of the GPU host, and proposes separating token parsing from GPU hosts.
Proposes framing AI evaluations as a red-team versus blue-team game in which a scheming model subverts evals via evaluation awareness, exploration hacking, or sandbagging, and analyses robustness versus richness of alignment and capability evals.
Argues that untrusted monitoring against collusion needs three countermeasures (honeypots, trusted monitoring, signal jamming), with red-team analysis showing each is necessary to prevent collusion between AI models.
Argues AI control research must expand to harness-level threats, demonstrating how Claude Code features like skills, subagents, memory and compaction let a rogue agent evade auto-mode monitoring and proposing vulnerability research and harness evaluations.
Shows on Llama-3.3-70B that activation components outside the model's self-reportable J-space still steer outputs, and that Anthropic's Natural Language Autoencoder reads them while the model denies them, with implications for oversight.
Proposes cryptographically chaining every sub-agent inference call to a verified human instruction via hardware MFA, so unmonitored rogue agents cannot delegate misaligned work to monitored agents undetected.
Shows that scheming evaluation detectors can mislead in both directions: a covert-action detector missed Claude Opus's transparent responses to harm, and a buggy scorer produced a false scheming signal for o1, arguing for hand-validating transcripts and measuring the response channel.
Measures how long a task frontier models can solve without chain of thought, finding no-CoT horizons doubling yearly, which matters for CoT monitoring and hidden scheming.
Revisits resampling and compares retrying protocols for AI control in BashArena, finding retrying with monitor feedback is exploitable and resampling aggregation choices matter for safety.
Argues that paraphrasing is at best a partial defence against LLM steganography in chain-of-thought, since semantic channels survive it and models can adapt to paraphrasers, and proposes better paraphrasing directions.
Tests whether SFT and RL can elicit the true performance of prompted sandbagging models on Olympiad math, finding the gains mostly come from degraded system-prompt following rather than a real aversion to sandbagging.
Introduces LinuxArena, 20 realistic software engineering environments with side tasks and sabotage trajectories for evaluating AI sabotage capabilities, monitors, and control protocols.
Introduces Refactor Arena, a configurable control setting where untrusted agents refactoring codebases may insert delayed vulnerabilities, with deterministic scorers and initial evals validating the setup.
Shows that attack selection (start and stop policies) in agentic BashArena control evals substantially lowers safety against a trusted monitor, so fixed-attack-policy evaluations may overestimate protocol robustness.
Experiments test whether Claude Opus 4.5 and other models can solve arithmetic side tasks latently using filler tokens or phrase repetition in an AI-control setup, finding modest uplift mainly for Opus 4.5.
Argues that AI control monitors meant to prevent harm must run synchronously and block actions before execution, since live failures like weight exfiltration cannot be undone after detection.
Hobbyist experiments fine-tune Kimi K2.5 to produce load-bearing obfuscation in internal chain-of-thought that evades a monitor, with limited, admittedly rushed evidence of success.
Extends untrusted monitoring for AI control, arguing trusted monitors are essential, collusion signals are hard to block, and capabilities need indirect evaluation for safety cases.
Studies collusion strategies against untrusted monitoring in AI control, finds SFT-elicited passive self-recognition beats prior attacks, and sketches a safety case with open gaps.
Introduces CoT-Control, a suite measuring whether reasoning models can control their chain-of-thought, finding low controllability that suggests CoT monitoring remains viable for now.
Lays out the structure of AI Control evaluations, covering monitor information channels, monitor capability, environments and side tasks, and proposes research workstreams for measuring control outcomes.
Fine-tuning GPT-4.1 on AI control literature raised blackmail rates in the Agentic Misalignment scenario, while monitor collusion was absent except for a brittle, eval-aware Grok 4 result.
Shows black-box LLM monitors optimized on synthetic scheming trajectories transfer to realistic sandbox environments, with performance saturating after a simple prompt selection over 15 candidates.
Proposes decomposing takeover risk in multi-model deployment into alignment, diversification, and control, with a formal model showing diversification helps only when the danger function is convex.
Trains LLM agents to call a report_scheming tool alongside side-task actions, so covert misbehavior becomes visible; it cuts undetected attacks versus blackbox monitors across 15 environments with minimal safety tax and generalizes to uninstructed misbehavior.
Shows that most chain-of-thought steps in LLMs are decorative rather than causally used, and that a linear latent direction can steer whether a step is internally engaged, undermining CoT monitoring.
Introduces MonitoringBench, a dataset of human-seeded and refined attack trajectories for testing AI control monitors, finding that only Opus-4.5 reliably catches strong attacks and that monitors often fail to calibrate suspiciousness.
A literature review finding that LLMs coordinate on simple Schelling tasks but only weakly on covert channels and control subversion, arguing risk lies in coordination tied to strategic goals and threatens untrusted monitoring.