Tokenizer Fertility Impacts Representation Reliability in Low-Resource Languages
September 30, 2026
Research shows sentiment direction reliability drops significantly in Hausa (0.589–0.762) and Yoruba (0.101–0.399) compared to English (0.737–0.870). The findings demonstrate that high classification accuracy does not guarantee directional consistency in model representations, often due to tokenizer fertility differences.
HOW THIS AFFECTS YOU
●
researcherYou must account for tokenizer-induced representation drift when evaluating multilingual model alignment or interpretability.