LLM Judge Panel Errors Concentrate on Pivotal Votes

Research shows that LLM judge panels exhibit high error correlation, with nine judges providing only the information equivalent of two independent ones. However, adding external evidence like test suites improves accuracy by 10.4 to 23.3 percentage points specifically on decisions with a one-vote margin.