Reward Model Scores Cluster by Prompt Rather Than Human Ratings
October 23, 2026
An audit of eighteen reward models reveals that scores cluster by prompt with between-prompt shares between 0.34 and 0.64, while human rating variance across prompts remains near zero. This indicates that reward models fail to capture prompt-independent quality components during pairwise evaluation.
HOW THIS AFFECTS YOU
●
builderThis suggests your reward-based ranking systems may be sensitive to prompt distribution rather than response quality.
●
researcherYou should account for prompt-level bias when evaluating the reliability of preference optimization models.