Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

Mooncake implements a disaggregated architecture for LLM serving by separating prefill and decoding clusters to optimize KVCache management. It utilizes CPU, DRAM, and SSD resources for a distributed cache, achieving up to 525% throughput increases in long-context simulations while maintaining latency SLOs through a prediction-based early rejection policy.