Byte-Level Transformers Outperform Subword Models at Scale

Byte-level Transformers, using tokenization-free training with token-superposition and hash embeddings, consistently outperform subword-based models as scale increases. The models learn to build internal text abstractions, achieving high performance even when intermediate layers are restricted to these learned segmentations.