The Functionalizer Provides Lossless Subword Tokenization via Opcodes

The Functionalizer is a pre-tokenizer that uses a compositional opcode/operand stream to handle orthographic variations like casing and diacritics. This approach prevents embedding space fragmentation and avoids the information loss associated with traditional normalization.