LLM Scoring Sensitivity Varies Significantly Based on Rating Scale Design
October 16, 2026
Research demonstrates that changing rating scale formats, ranges, or signs shifts LLM-generated scores by an average of 0.10 in normalized values. The study evaluates numerical and semantic sensitivity across five LLMs using various continuous and ordinal scales for tasks like sentiment and toxicity analysis.
HOW THIS AFFECTS YOU
●
builderYou should be cautious when using LLMs for automated scoring, as scale design introduces systematic bias.
●
researcherYou must standardize your rating scale designs to ensure comparability across different LLM evaluations.