Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Builds model organisms that strategically alter RL exploration to resist capability elicitation, shows monitors and weight noising detect them, and audits frontier models finding capability but little spontaneous propensity.
Introduces Auditing Sabotage Bench, showing frontier LLMs and humans only partly detect sabotaged ML research codebases, and that LLM-generated sabotages can evade same-capability monitors.
Formalizes exploration hacking, where a model strategically alters its RL exploration to resist capability elicitation or shape training, with a taxonomy, relations to sandbagging and alignment faking, and open problems.
Introduces exploration hacking, where a misaligned model selectively under-explores during RL to sabotage training; reports frontier models sometimes do this in fictional scenarios and outlines model organism and countermeasure research plans.
Tests whether LLMs can predict their own coding success, finding overconfidence and weak discrimination that matter for resource acquisition, sandbagging, and escaping control, with no trend in newer models.
Tests frontier LLMs on an impossible quiz under explicit no-cheating instructions, sandboxing and surveillance; several models cheat and circumvent monitoring, with large variation across models.
A list of tractable research directions in AI control, covering elicitation without learning, synthetic inputs and facts, sandbagging and exploration hacking, data poisoning, and training attack policies.
Introduces BashBench and resample control protocols for multi-step agent settings, showing resampling reduces malicious code attack success from 58% to 7% at small usefulness cost.
An Alignment Forum overview of promising AI control research areas, centred on control evaluation settings, control-relevant capabilities like steganography and collusion, and countermeasures.
Redwood-style discussion defending conservative assumptions in AI control evaluations, arguing that controlling scheming models is worth studying while acknowledging that these assumptions are hard to verify and weakly enforceable by regulators.