Analysis Reveals Unreliable LLM-as-a-Judge Performance in Multilingual Settings

An investigation of 650 papers shows that LLM-based evaluation is inconsistent and frequently overtrusted in low-resource and multilingual contexts. Current practices lack adequate human validation, leading to potential biases in how non-English model performance is measured.