Analysis of Representation Convergence in Transformer Residual Streams

Study of six pretrained models shows that intermediate residual states align with their own final states early in the depth process. While directional alignment and endpoint rank improve, Euclidean distance to the final state remains relatively stable.