GenRM Performance Divergence Between Static Accuracy and Online Policy Training
October 23, 2026
Thinking-based reward models achieve higher static accuracy in Qwen3 (0.6B to 14B) using GRPO, but Answer-Only models yield superior policy Elo during online DPO training. Results show that larger reward models, including those exceeding policy size, provide significant benefits for policy optimization.
HOW THIS AFFECTS YOU
●
builderYou should consider using Answer-Only rewarders if your primary goal is maximizing policy Elo via online DPO.
●
researcherThis highlights a disconnect between static gold-label accuracy and functional utility in reinforcement learning loops.