Reference Choice Impact on Multilingual Benchmark Scores
October 1, 2026
Analyzing a six-language PII benchmark reveals that changing the gold reference can shift F1 scores by up to 7.55 points. This demonstrates that benchmark results are highly sensitive to the choice of reference rather than just model performance.
HOW THIS AFFECTS YOU
●
researcherYou should report confidence intervals for benchmark scores to account for reference-induced variance.
●
policyRegulatory assessments relying on single-reference benchmarks may misrepresent model safety performance.