Circuit Analysis of In-Context Learning Mechanisms

This study identifies a key-value memory mechanism in 3-4B parameter transformers that enables in-context learning. The model uses label-token positions as memory, where middle-layer attention heads write class information into the residual stream for upper-layer readout.