Depth Growth and Looping Increase In-Context Learning Retention

Analysis of models up to 3B parameters shows that depth growth and looping architectures improve in-context learning (ICL) acquisition and retention. These architectures propagate induction heads across layers more effectively than full-depth dense models, relying more heavily on contextual information.