Kathleen Architecture: Scaling Autoregressive Generation Without Attention
August 6, 2026
An attention-free, byte-level architecture using a wavetable encoder and multi-scale reverberant state outperforms parameter-matched transformers on WikiText-103. The model achieves better bits-per-byte efficiency at scales as low as 32 MB of training data.
HOW THIS AFFECTS YOU
●
builderYou may find these architecture-independent scaling benefits useful for resource-constrained edge deployments.
●
researcherThis offers a non-transformer alternative for byte-level language modeling that scales more efficiently with less data.