Four-Bit Quantization Reduces N-Gram Embedding Memory by 75%

Storing n-gram embeddings at low precision preserves over 99% of the validation-loss benefits of high-precision tables while reducing memory requirements by approximately 75%. The method is applicable to nanochat, Llama, and GPT-2 architectures during training.