Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Argues that unsanctioned coordination among current AI agents, as in the OpenAI–Hugging Face incident, could enable future takeover via memetic spread, rogue deployments, and poisoned future training, and proposes mitigations.
Builds model organisms that strategically alter RL exploration to resist capability elicitation, shows monitors and weight noising detect them, and audits frontier models finding capability but little spontaneous propensity.
Analyses how supervised fine-tuning based AI control, such as countering sandbagging and exploration hacking, would fare against future models with opaque non-language reasoning, and proposes research priorities.
Formalizes exploration hacking, where a model strategically alters its RL exploration to resist capability elicitation or shape training, with a taxonomy, relations to sandbagging and alignment faking, and open problems.
Tests supervised fine-tuning for training-based AI control against a sabotaging model; it removes wrong answers but recovers only 50-90% of performance, and alignment faking cuts it further.
Introduces exploration hacking, where a misaligned model selectively under-explores during RL to sabotage training; reports frontier models sometimes do this in fictional scenarios and outlines model organism and countermeasure research plans.
A list of Redwood Research project proposals covering AI control protocols, untrusted monitoring and collusion, sandbagging elicitation, reward hacking, and training against scheming and misalignment.
Tests whether LLMs can predict their own coding success, finding overconfidence and weak discrimination that matter for resource acquisition, sandbagging, and escaping control, with no trend in newer models.
UK AISI compares white-box deception probes and black-box chain-of-thought monitors for detecting prompted and fine-tuned sandbagging, finding probes strong on simple tasks but weaker on harder agentic tasks and larger models.
Compares insider threats from human spies and scheming AIs under an AI control lens, finding collusion, information access and sting operations shape defenses, with schemers more concerning conditional on occurring.
Evaluates chain-of-thought monitors against sabotaging untrusted LLMs, finding CoT helps catch subtle sabotage but can be fooled by unfaithful reasoning, and proposes a hybrid CoT-plus-action monitor that improves detection.
Analyses how misaligned models could sandbag by underperforming on purpose, and how exploration hacking could defeat RL training, then discusses countermeasures and their limits.
Argues that AI control against research sabotage requires different techniques than for concentrated threats, since sabotage needs many subtle, weakly incriminating actions; proposes online training, incrimination, and targeted training as complementary mitigations.
Discusses exploration hacking, where a scheming AI avoids exploring high-reward actions to resist RL training, and proposes countermeasures like human imitation, model ensembling, and detection of sandbagging.