LLM Robustness Varies Across 27 Languages Under Non-Canonical Tokenization
July 30, 2026
LLM performance is not invariant to alternative tokenizations in non-English languages. A study of 27 languages across six tasks shows that instruction-tuned models exhibit significant performance shifts when encountering non-canonical token sequences, unlike the relative stability observed in English.
HOW THIS AFFECTS YOU
●
builderYou may need to implement more robust preprocessing for non-English user inputs to maintain consistent model behavior.
●
researcherYou should investigate how tokenization-induced variance affects multilingual model evaluation.