Phonemic Tokenizer Factorizes Syllables for Chinese and Vietnamese
September 21, 2026
The Phonemic Tokenizer replaces subword tokens with three phonological components: onset, rime, and tone. Using IPA factorization, it achieves high Rènyi efficiency with extremely small vocabularies of only 112 entries for Chinese and 256 for Vietnamese.
HOW THIS AFFECTS YOU
●
researcherYou can improve representation sharing in multilingual models by moving beyond statistical subwords to phonological structures.