Research / Sources
Understanding and steering neural networks. goodfire.com
Introduces mechanistic shaping, which trains models through their internal representations to be right for the right reasons, reducing sycophancy, reward hacking and confounder use, and studies effects on monitorability.
Predicts which behaviours DPO will amplify or suppress from a preference dataset before training, traces them to responsible data, and proposes reshaping the data to prevent undesired effects.
Uses probes to trace a harmful compliance behavior that emerges during DPO post-training of OLMo 2 to specific preference datapoints, then filtering or relabeling them cuts it substantially.
Goodfire trains Gemma-3-12B with RL using lightweight probes on internal activations as rewards (RLFR), cutting hallucinations by 58% without benchmark degradation, while the probe remains usable as a test-time monitor.
Goodfire essay arguing that interpretability should guide AI training so new models share human values, framing intentional design as a scientific alternative to opaque training.