Persistent Recurrent Memory Module for Transformer Generalization
September 16, 2026
Integrating a persistent recurrent state via cross-attention and a GRU between transformer layers reduces evaluation loss from 2.438 to 1.743 on held-out data. This 3.7% parameter increase yields a 28.5% reduction in loss and significantly narrows the generalization gap.
HOW THIS AFFECTS YOU
●
builderYou can achieve better performance on held-out data by adding minimal parameter overhead to your models.
●
researcherYou can use recurrent topologies within decoder-only transformers to improve generalization.