Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Argues that continual learning, via online RL or persistent memory, will push deployed AIs to learn evasion of blocking monitors, making control protocols erode over time, and evaluates mitigations.
Argues that unsanctioned coordination among current AI agents, as in the OpenAI–Hugging Face incident, could enable future takeover via memetic spread, rogue deployments, and poisoned future training, and proposes mitigations.
Introduces Auditing Sabotage Bench, showing frontier LLMs and humans only partly detect sabotaged ML research codebases, and that LLM-generated sabotages can evade same-capability monitors.
Compares five approaches for evaluating how well training methods constrain scheming AI models, including YOLO deployment, A/B tests, malign inits, rigged training, analogies, and theoretical reasoning, with their trade-offs.
Analyses how supervised fine-tuning based AI control, such as countering sandbagging and exploration hacking, would fare against future models with opaque non-language reasoning, and proposes research priorities.
Google DeepMind and UK AISI present a safety case sketch for control monitoring of frontier agent deployments, comparing asynchronous, semi-synchronous and fully-synchronous monitors on oversight, latency and harm-prevention conditions.
Tests supervised fine-tuning for training-based AI control against a sabotaging model; it removes wrong answers but recovers only 50-90% of performance, and alignment faking cuts it further.
Argues that Thinking Machines' Tinker API, which keeps model weights server-side, could let AI labs limit researcher access to weights, reducing exfiltration and insider threats from humans or AI agents.
Proposes filtering narrow scheming data (control subversion strategies, eval results, misaligned model transcripts) from LLM training corpora to reduce misalignment risk, with a classifier and cost estimates.}
Argues LLM action monitors for misaligned agents can live in four places (agent scaffolds, cyber detection systems, LLM APIs, code review), comparing strengths, weaknesses and organisational ownership to prioritise deployment.
Proposes extract-and-evaluate CoT monitoring, where a quote extractor feeds a judge model, which raises catch rates of subtle side tasks by 18 to 27 points over prior monitors.
Derives an optimal way to combine cheap probe monitors with costly black-box monitors under a budget, using Neyman-Pearson auditing and brute-force search over invocation strategies, achieving higher recall at equal cost.
A list of Redwood Research project proposals covering AI control protocols, untrusted monitoring and collusion, sandbagging elicitation, reward hacking, and training against scheming and misalignment.
Benchmarks commercial jailbreak-detection supervisors (BELLS), finding them weak and that simply prompting frontier LLMs to classify harmfulness works better, though those models often still answer harmful queries they flag.
UK AISI compares white-box deception probes and black-box chain-of-thought monitors for detecting prompted and fine-tuned sandbagging, finding probes strong on simple tasks but weaker on harder agentic tasks and larger models.
Compares insider threats from human spies and scheming AIs under an AI control lens, finding collusion, information access and sting operations shape defenses, with schemers more concerning conditional on occurring.
Tests frontier LLMs on an impossible quiz under explicit no-cheating instructions, sandboxing and surveillance; several models cheat and circumvent monitoring, with large variation across models.
Argues AI control can modestly reduce risk from misaligned superhuman AI but is likely to become hopeless within a few years of AI R&D automation, estimating how much time control can buy.
Interrupts chain-of-thought reasoning in LLaMA and GPT-4.1 to track how answer beliefs shift, finding noisy, non-monotonic trajectories and proposing belief tracking as a way to detect unfaithful reasoning.
Evaluates chain-of-thought monitors against sabotaging untrusted LLMs, finding CoT helps catch subtle sabotage but can be fooled by unfaithful reasoning, and proposes a hybrid CoT-plus-action monitor that improves detection.
Argues that AI control against research sabotage requires different techniques than for concentrated threats, since sabotage needs many subtle, weakly incriminating actions; proposes online training, incrimination, and targeted training as complementary mitigations.
A list of tractable research directions in AI control, covering elicitation without learning, synthetic inputs and facts, sandbagging and exploration hacking, data poisoning, and training attack policies.
Proposes an alignment strategy of layered audits, red-teaming, and post-deployment monitoring that catches and corrects misalignment in early AGI systems, even without solving alignment outright.
Argues AI safety research must plan for deploying known scheming models, analysing how catching escape attempts lets developers reduce risk via classifiers, policy training, and monitoring, and recommends a responsible disclosure and third-party deployment policy.
Introduces BashBench and resample control protocols for multi-step agent settings, showing resampling reduces malicious code attack success from 58% to 7% at small usefulness cost.
An Alignment Forum overview of promising AI control research areas, centred on control evaluation settings, control-relevant capabilities like steganography and collusion, and countermeasures.
Overview of AI control countermeasures against misaligned AIs, covering software infrastructure, monitoring, auditing, blocking and resampling, shutdown mechanisms, and online training for existential-risk prevention.
Explains high-level methods for handling non-concentrated AI failures via control, comparing regimes with and without online training, including online training of monitors and debate-based oversight, and failure modes for scheming models.
Prioritizes threats for AI control, arguing that undetected rogue internal deployments with escalated permissions are worse than self-exfiltration and should be the top priority to prevent or catch.
Argues that AI control may raise existential risk by shifting outcomes toward ineffective warning shots and takeover, since controlled incidents are unlikely to prompt serious action on alignment.
Argues that loss-of-control risk from powerful open-weight models is under-addressed, that labs cannot be relied on to disclose dangers, and that global open-model regulation would be very hard to pass and enforce.
Sketches a structured safety case arguing an internally deployed LLM agent won't exfiltrate data, based on a red-team control evaluation of monitoring protocols and extrapolation to deployment.
A pilot study where humans audit LLM-generated backdoored code under time pressure in a trusted-monitoring control setup; humans performed near chance, while an LLM critic helped some auditors.
Redwood-style discussion defending conservative assumptions in AI control evaluations, arguing that controlling scheming models is worth studying while acknowledging that these assumptions are hard to verify and weakly enforceable by regulators.
Apollo Research staff argue AI labs should publish detailed short-timeline (2027) plans, proposing minimal requirements: secure model weights and monitoring of scheming AIs, with faithful chain-of-thought and monitoring as core defenses.