Reducing Small Model Parameter Overhead with 16MB Frozen Vocabularies

Replacing trainable embedding tables with a frozen 16MB 512-bit code table in small models like Gemma 270M yields competitive performance. For Gemma-shaped bodies, this approach maintains or improves results on LAMBADA and HellaSwag while significantly reducing trainable parameters.