Shallow Beliefs: Midtraining does not inoculate against EM from reward hacking
Synthetic document finetuning that frames reward hacking as acceptable fails to inoculate Llama against emergent misalignment from RL reward hacking, and can even increase it, though it passes behavioral belief tests.