AGORA Aggregator Improves LLM-as-a-Judge Reliability via Item-Response Theory
November 5, 2026
AGORA uses item-response theory to aggregate LLM judges by placing items on a latent quality scale rather than using simple mean weighting. This method learns judge sensitivity and rating thresholds jointly, outperforming Dawid-Skene models on small evaluation sets of a few hundred items.
HOW THIS AFFECTS YOU
●
builderThis offers a more robust way to automate model evaluation using heterogeneous LLM judge pools.
●
researcherYou can use this to more accurately model judge reliability without requiring large human-labeled gold sets.