Token-Based Models Outperform Byte-Based Models in Large-Scale Distillation Studies

A study of models up to 1 trillion bytes shows that token-based models outperform byte-based variants (End-Of-Token and Bytes-with-EOT) across multiple benchmarks in QA, generation, and translation. The researchers introduced Marginalize-It and End-Of-Token methods for converting token logits to byte logits.