Chatbot Arena Bradley-Terry Model Fails to Distinguish Similar Models
October 15, 2026
Analysis of 1.8 million Chatbot Arena votes shows that the Bradley-Terry model underestimates the decisiveness of votes between similar models. A new correction model better predicts outcomes by accounting for ties and model similarity in rankings.
HOW THIS AFFECTS YOU
●
researcherYou should account for model similarity and tie-handling when evaluating your model's ranking in Arena.