Layerwise Geometric Progression of Relational Representations in LLMs

Analysis of Llama, Qwen, Gemma, and Mistral models shows a consistent progression where intermediate layers process relationships between two tokens, while later layers utilize joint representations of three or more tokens for cyclic tasks. This geometric organization reveals how transformer architectures causally manage relational information across depth.