Rubric Language and Clause Structure Affect LLM Judge Reliability

Evaluation of Korean support dialogues shows that LLM judges exhibit lower cross-provider agreement when using Korean rubrics compared to English ones. Disagreements are most concentrated in negated, conditional, and exception clause structures, impacting the consistency of automated quality assurance.