LLM judges show scale-direction bias in benchmark evaluations
November 20, 2026
Testing seven LLM judges reveals widespread violations of semantic faithfulness, where models respond to numerical scale anchors rather than rubric meaning. The study introduces floor normalization and bidirectional anchor averaging to mitigate this bias.
HOW THIS AFFECTS YOU
●
builderDon't rely on raw LLM scores for automated benchmarking without checking for scale-direction bias.
●
researcherYou should audit your LLM-based evaluators for scale-reversal bias to ensure valid results.