Discussion and research on AI alignment. alignmentforum.org
Argues funders should make large scaling grants for automated AI safety labor, such as automated monitoring, black-box and white-box auditing, propensity evals, and automated conceptual alignment research.
A funding call outlining cognitive-science research directions for AI safety: reducing human rater error in alignment, testing scalable oversight protocols with human participants, and human auditing for AI control.
Proposes a prefix-cache untrusted monitor, a copy of the policy fine-tuned to judge its own recent actions while reusing its KV cache, as an alternative to training the scheming policy after catching egregious failures.</parameter>
Argues that AI safety work, including evals, red-teaming, and trusted monitoring, should be automated with current AI capabilities, and proposes concrete pipelines and preparation steps for doing so.
Explains high-level methods for handling non-concentrated AI failures via control, comparing regimes with and without online training, including online training of monitors and debate-based oversight, and failure modes for scheming models.
A pilot study where humans audit LLM-generated backdoored code under time pressure in a trusted-monitoring control setup; humans performed near chance, while an LLM critic helped some auditors.