Grouped Query Attention: Reducing Inference Costs

Grouped-Query Attention (GQA) is a technique that interpolates between Multi-Head and Multi-Query Attention to reduce KV cache size and memory footprint during inference while preserving model quality.