AdaptTok-Indic Reduces Token Fertility for Indic Languages
October 27, 2026
AdaptTok-Indic utilizes script-aware routing and learned byte-level boundary prediction to improve tokenization for Bengali and Hindi. The method achieves mean fertility rates of 1.432 for Bengali and 1.388 for Hindi, significantly outperforming SentencePiece BPE which recorded 2.156 and 2.085 respectively.
HOW THIS AFFECTS YOU
●
builderYou can reduce inference costs and sequence length when processing Indic language datasets.
●
researcherThis method provides a specialized baseline for studying script-aware tokenization improvements.