Decidability Framework for Finitely Verifiable Tokenizer Segmentation

This research introduces a class of segmentation policies that are finitely verifiable via a finite-state automaton. It enables decidability for four key practical questions: boundary certification, boundary gap bounds, mask-based generation completion, and policy equivalence.