BPE and Unigram-LM Tokenizer Discrepancy in Chemistry SMILES
July 5, 2026
A controlled comparison shows that BPE and Unigram-LM produce near-disjoint subword vocabularies when tokenizing SMILES strings, with Jaccard overlap never exceeding 0.161. The results demonstrate that these algorithms do not converge even when using a fixed 165-token chemistry base.
HOW THIS AFFECTS YOU
●
builderYou should carefully evaluate tokenizer impact on your SMILES-based model performance.
●
researcherThis highlights the need for domain-specific tokenizer selection in chemical language modeling.