More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding

Sparse Asymmetric Group-Query Attention (SAGA) decouples key and value head counts to reduce query-key computational bottlenecks during LLM decoding. Combined with approximate top-N attention, this method allows for reduced key heads while maintaining value head capacity to preserve model expressivity without increasing latency.