Amortizing In-Context Learning into Latent Task Representations

A 2.6M-parameter network extracts the geometry of few-shot support sets to produce additive residual stream updates in frozen GPT-2 models. This allows for zero-shot inference of specific linguistic operations, such as forward inflection, by replacing standard in-context learning with cached latent vectors.