Discussion and research on AI alignment. alignmentforum.org
Argues that continual learning, via online RL or persistent memory, will push deployed AIs to learn evasion of blocking monitors, making control protocols erode over time, and evaluates mitigations.
Compares five approaches for evaluating how well training methods constrain scheming AI models, including YOLO deployment, A/B tests, malign inits, rigged training, analogies, and theoretical reasoning, with their trade-offs.
Proposes grounding AI value learning in evolutionary psychology as an alternative to CEV, arguing AI should be treated as an engineered tool to be aligned rather than a moral patient.
Proves information-theoretic lower bounds on alignment cost, including inevitable reward hacking, and gives formal multi-step corrigibility guarantees via lexicographically ordered utility heads.
Proposes filtering AI-misalignment discussion from pretraining data to reduce hyperstitional misalignment risk, weighing costs to safety research and deployment, and argues labs should weakly adopt it.
An exploratory essay that maps RL policy entropy and RLHF onto psychotherapy concepts, reviews RLHS and RISE, and proposes three untested alignment training methods.
Steve Byrnes analyses 'reward button alignment', where an AGI is trained to want a reward button pressed, finding it could work for a while but would likely fail through power-seeking, and assesses bootstrapping and debate variants.
Analyses how misaligned models could sandbag by underperforming on purpose, and how exploration hacking could defeat RL training, then discusses countermeasures and their limits.
Argues that training on interpretability tools is useful if done once and kept last, but that running capabilities RL after RLHF or Constitutional AI lets hidden misbehaviour re-emerge.
Discusses exploration hacking, where a scheming AI avoids exploring high-reward actions to resist RL training, and proposes countermeasures like human imitation, model ensembling, and detection of sandbagging.
Explains high-level methods for handling non-concentrated AI failures via control, comparing regimes with and without online training, including online training of monitors and debate-based oversight, and failure modes for scheming models.