Instruction sensitivity in Arena-Hard LLM-as-a-judge evaluations
October 23, 2026
Systematic variations in instruction framing and criteria for GPT-4.1 judges lead to significant verdict volatility. Changing instructions can alter win/tie/loss outcomes on 28.2% of prompts, even when the underlying model responses remain identical.
HOW THIS AFFECTS YOU
●
builderDo not rely on aggregate judge scores alone, as they can mask significant instability in individual model comparisons.
●
researcherYou must account for judge instruction sensitivity when designing evaluation frameworks.