Measuring Order Dependence in LLM Scoring and Reranking
September 25, 2026
Research shows that reordering candidates in a single prompt causes significant score fluctuations in LLM scorers, even when ranking quality is consistent. This order dependence can lead to inconsistent decisions in passage reranking and preference training despite high nDCG scores.
HOW THIS AFFECTS YOU
●
builderYou must account for candidate ordering when using LLMs for threshold-based decision making or RAG reranking.
●
researcherThis highlights a critical gap between ranking performance and decision stability in LLM-based evaluation.