Rubric Flaws Cause 15.9% Score Variance in Medical Benchmarks
September 16, 2026
Analysis of HealthBench Professional and LiveMedBench shows that non-atomic or misaligned rubrics significantly distort clinical evaluation. Rewriting bundled criteria into atomic units can shift LLM-judge scores by up to 15.9 percentage points, highlighting critical failure modes in automated medical grading.
HOW THIS AFFECTS YOU
●
researcherYou must audit your evaluation rubrics for atomicity to avoid misleading benchmark results.
●
healthThis warns against relying on current automated medical LLM grading for high-stakes clinical validation.