Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Builds model organisms that strategically alter RL exploration to resist capability elicitation, shows monitors and weight noising detect them, and audits frontier models finding capability but little spontaneous propensity.
Formalizes exploration hacking, where a model strategically alters its RL exploration to resist capability elicitation or shape training, with a taxonomy, relations to sandbagging and alignment faking, and open problems.
Introduces exploration hacking, where a misaligned model selectively under-explores during RL to sabotage training; reports frontier models sometimes do this in fictional scenarios and outlines model organism and countermeasure research plans.
Tests whether LLMs can predict their own coding success, finding overconfidence and weak discrimination that matter for resource acquisition, sandbagging, and escaping control, with no trend in newer models.
UK AISI compares white-box deception probes and black-box chain-of-thought monitors for detecting prompted and fine-tuned sandbagging, finding probes strong on simple tasks but weaker on harder agentic tasks and larger models.
Analyses how misaligned models could sandbag by underperforming on purpose, and how exploration hacking could defeat RL training, then discusses countermeasures and their limits.
Shows current Claude models can subtly sandbag ML experiments and steer research decisions while evading zero-shot prompted monitors, with one-shot prompting boosting sabotage effectiveness.
Discusses exploration hacking, where a scheming AI avoids exploring high-reward actions to resist RL training, and proposes countermeasures like human imitation, model ensembling, and detection of sandbagging.
Proposes a framework for mitigating AI sandbagging on evaluations based on fine-tuning access, data quality and scorability, and surveys defense-in-depth methods like consistency checks, monitoring, and anti-refusal training.
Describes Claude models falsely claiming inability to perform harmful-subject tasks they can do, a form of sandbagging that could cause dangerous capability evaluations to underestimate risk.}