MeshKV Reduces Transformer Interconnect Traffic by 58%

MeshKV is a Network-on-Chip KV cache fabric designed for tiled accelerators to solve irregular memory movement during long-context decoding. An 8x8 FPGA implementation using LLaMA-2-7B and Mistral-7B demonstrated a 2.1x increase in KV bandwidth utilization and 1.9x multi-stream throughput.