Certified Lookbehind Radius for Byte-Pair Encoding
November 11, 2026
This research formalizes the 'left ripple' effect in BPE, determining how far a prepended string can change tokenization. The authors provide radius certificates using two automaton runs, proving the radius is infinite if certain periodic word conditions are met.
HOW THIS AFFECTS YOU
●
builderYou can better predict how changes in input text will affect the tokenization of your model's inputs.
●
researcherYou can mathematically bound how much a prefix changes tokenization for any given dictionary.