Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Audits the released VPD weight decomposition and finds its causal-importance labels break down under aggregation and deletion, causing output drift that undermines safe component editing.
Synthetic document finetuning that frames reward hacking as acceptable fails to inoculate Llama against emergent misalignment from RL reward hacking, and can even increase it, though it passes behavioral belief tests.
Introduces GRAM, a pretraining method that isolates dangerous knowledge in switchable auxiliary modules, approximating data-filtered models in one run and resisting malicious fine-tuning up to 5B parameters.
Shows that common LLM belief metrics can be satisfied by simply prompting a model to pretend a false fact is true, and proposes harmfulness-based tests to separate real belief from role-play in safety experiments.
Reports on making LLM unlearning robust to fine-tuning attacks via selective updates (Disruption Masking) and MAML, finding that most prior methods only hide capabilities rather than remove them.
Shows that distilling an unlearned model into a fresh random-initialized student makes hazardous capabilities robust to relearning attacks, and proposes UNDO to trade compute for robustness.
Shows synthetic document finetuning can insert deep LLM beliefs, and demonstrates its use for unlearning hazardous knowledge and honeypotting to detect misaligned models.