Gating Mechanisms Impede In-Context Learning in State Space Models

Analysis of State Space Models (SSMs) reveals that gating mechanisms drive models toward in-weights memorization, which delays or prevents convergence to in-context learning solutions. This architectural behavior explains why SSMs often lag behind Transformers in precise retrieval and few-shot tasks.