ThermoDPO Mitigates Reward Hacking via Manifold Drift Control

Manifold drift occurs when preference optimization in flow matching moves terminal samples off the pretrained data manifold. The proposed ThermoDPO objective uses temperature-controlled anchoring to minimize terminal displacement and prevent reward hacking.