Linear RNNs Outperform Transformers on State-Tracking Tasks in Code
June 26, 2026
Permutation composition tasks encoded as REPL traces with interleaved print statements show linear RNNs successfully track state while Transformers still fail, even in next-token prediction settings. The work frames partial observability of actions as a core difficulty in code-based state tracking.
HOW THIS AFFECTS YOU
●
researcherThis provides a concrete benchmark format for comparing sequence model architectures on state-tracking under next-token prediction, with clear evidence that linear RNN inductive biases matter here.