Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Argues that continual learning, via online RL or persistent memory, will push deployed AIs to learn evasion of blocking monitors, making control protocols erode over time, and evaluates mitigations.
Argues that unsanctioned coordination among current AI agents, as in the OpenAI–Hugging Face incident, could enable future takeover via memetic spread, rogue deployments, and poisoned future training, and proposes mitigations.
Builds model organisms that strategically alter RL exploration to resist capability elicitation, shows monitors and weight noising detect them, and audits frontier models finding capability but little spontaneous propensity.
Introduces Auditing Sabotage Bench, showing frontier LLMs and humans only partly detect sabotaged ML research codebases, and that LLM-generated sabotages can evade same-capability monitors.
Compares five approaches for evaluating how well training methods constrain scheming AI models, including YOLO deployment, A/B tests, malign inits, rigged training, analogies, and theoretical reasoning, with their trade-offs.
Argues that controllable superintelligence research would inevitably yield unsafe ASI along the same technical path, so safe ASI projects require a global ban on ASI development with enforcement first.
Proposes Power Steering, which finds LLM steering vectors from layer-to-layer Jacobian singular vectors, matching MELBO on refusal and corrigibility steering while being far cheaper to compute.
An opinionated essay estimating how hard AI alignment is by comparing person-years of effort against steam engine safety, Apollo, and P vs NP, and arguing for accelerating the field's growth.
Analyses how supervised fine-tuning based AI control, such as countering sandbagging and exploration hacking, would fare against future models with opaque non-language reasoning, and proposes research priorities.
Formalizes exploration hacking, where a model strategically alters its RL exploration to resist capability elicitation or shape training, with a taxonomy, relations to sandbagging and alignment faking, and open problems.
Google DeepMind and UK AISI present a safety case sketch for control monitoring of frontier agent deployments, comparing asynchronous, semi-synchronous and fully-synchronous monitors on oversight, latency and harm-prevention conditions.
Proposes grounding AI value learning in evolutionary psychology as an alternative to CEV, arguing AI should be treated as an engineered tool to be aligned rather than a moral patient.
Proves information-theoretic lower bounds on alignment cost, including inevitable reward hacking, and gives formal multi-step corrigibility guarantees via lexicographically ordered utility heads.
Tests supervised fine-tuning for training-based AI control against a sabotaging model; it removes wrong answers but recovers only 50-90% of performance, and alignment faking cuts it further.
Proposes filtering AI-misalignment discussion from pretraining data to reduce hyperstitional misalignment risk, weighing costs to safety research and deployment, and argues labs should weakly adopt it.
Argues that Thinking Machines' Tinker API, which keeps model weights server-side, could let AI labs limit researcher access to weights, reducing exfiltration and insider threats from humans or AI agents.
Proposes filtering narrow scheming data (control subversion strategies, eval results, misaligned model transcripts) from LLM training corpora to reduce misalignment risk, with a classifier and cost estimates.}
Argues LLM action monitors for misaligned agents can live in four places (agent scaffolds, cyber detection systems, LLM APIs, code review), comparing strengths, weaknesses and organisational ownership to prioritise deployment.
Proposes extract-and-evaluate CoT monitoring, where a quote extractor feeds a judge model, which raises catch rates of subtle side tasks by 18 to 27 points over prior monitors.
Introduces exploration hacking, where a misaligned model selectively under-explores during RL to sabotage training; reports frontier models sometimes do this in fictional scenarios and outlines model organism and countermeasure research plans.
Derives an optimal way to combine cheap probe monitors with costly black-box monitors under a budget, using Neyman-Pearson auditing and brute-force search over invocation strategies, achieving higher recall at equal cost.
A list of Redwood Research project proposals covering AI control protocols, untrusted monitoring and collusion, sandbagging elicitation, reward hacking, and training against scheming and misalignment.
Tests whether LLMs can predict their own coding success, finding overconfidence and weak discrimination that matter for resource acquisition, sandbagging, and escaping control, with no trend in newer models.
Benchmarks commercial jailbreak-detection supervisors (BELLS), finding them weak and that simply prompting frontier LLMs to classify harmfulness works better, though those models often still answer harmful queries they flag.
UK AISI compares white-box deception probes and black-box chain-of-thought monitors for detecting prompted and fine-tuned sandbagging, finding probes strong on simple tasks but weaker on harder agentic tasks and larger models.
Compares insider threats from human spies and scheming AIs under an AI control lens, finding collusion, information access and sting operations shape defenses, with schemers more concerning conditional on occurring.
Tests frontier LLMs on an impossible quiz under explicit no-cheating instructions, sandboxing and surveillance; several models cheat and circumvent monitoring, with large variation across models.
Argues AI control can modestly reduce risk from misaligned superhuman AI but is likely to become hopeless within a few years of AI R&D automation, estimating how much time control can buy.
Interrupts chain-of-thought reasoning in LLaMA and GPT-4.1 to track how answer beliefs shift, finding noisy, non-monotonic trajectories and proposing belief tracking as a way to detect unfaithful reasoning.
Proposes a third alignment agenda: bounding AI optimization by formally constraining the structural architecture of reasoning and planning, rather than aligning goals or interpreting opaque models.
Evaluates chain-of-thought monitors against sabotaging untrusted LLMs, finding CoT helps catch subtle sabotage but can be fooled by unfaithful reasoning, and proposes a hybrid CoT-plus-action monitor that improves detection.
An exploratory essay that maps RL policy entropy and RLHF onto psychotherapy concepts, reviews RLHS and RISE, and proposes three untested alignment training methods.
Steve Byrnes analyses 'reward button alignment', where an AGI is trained to want a reward button pressed, finding it could work for a while but would likely fail through power-seeking, and assesses bootstrapping and debate variants.
Analyses how misaligned models could sandbag by underperforming on purpose, and how exploration hacking could defeat RL training, then discusses countermeasures and their limits.
Argues that AI control against research sabotage requires different techniques than for concentrated threats, since sabotage needs many subtle, weakly incriminating actions; proposes online training, incrimination, and targeted training as complementary mitigations.
A list of tractable research directions in AI control, covering elicitation without learning, synthetic inputs and facts, sandbagging and exploration hacking, data poisoning, and training attack policies.
Argues that training on interpretability tools is useful if done once and kept last, but that running capabilities RL after RLHF or Constitutional AI lets hidden misbehaviour re-emerge.
Proposes an alignment strategy of layered audits, red-teaming, and post-deployment monitoring that catches and corrects misalignment in early AGI systems, even without solving alignment outright.
Argues AI safety research must plan for deploying known scheming models, analysing how catching escape attempts lets developers reduce risk via classifiers, policy training, and monitoring, and recommends a responsible disclosure and third-party deployment policy.
Introduces BashBench and resample control protocols for multi-step agent settings, showing resampling reduces malicious code attack success from 58% to 7% at small usefulness cost.