Decomposing Transformer Representation Evolution via Directional Geometry

Transformer updates are analyzed as functional geometry by decomposing learned transformations into parallel and perpendicular components relative to the hidden state. Findings show that exclude-self value-space parallel manipulation is more robust for preserving direct self-messages while scaling non-self aggregates.