Autonomy-of-Heads Enables Data-Free Sparse Attention via Spectral Geometry

Autonomy-of-Heads identifies retrieval and streaming heads by analyzing the effective rank of query-key projection spectral geometry. This data-free method allows for sparse attention and KV-cache compression without relying on costly runtime attention scores or calibration prompts.