Prompt Lottery Phenomenon in LLM-as-a-Judge Evaluations
October 30, 2026
The Prompt Lottery study reveals that minor syntactic rephrasings of judge prompts can alter evaluation outcomes and reverse model rankings. This instability challenges the reliability of using LLMs as standardized evaluation instruments for benchmarking other models.
HOW THIS AFFECTS YOU
●
builderYou should avoid relying on a single judge prompt for production-grade model evaluation.
●
researcherYou must account for syntactic instability when using LLMs to validate new model architectures.