llama.cpp implements heatmap-based MoE expert caching for GPU VRAM
August 4, 2026
A new llama.cpp pull request introduces a heatmap to cache frequently used MoE experts in VRAM while offloading others to CPU. On Qwen2.5-32B-A14B with 8GB VRAM, this achieved a 1.68x to 2.07x speedup, though performance gains depend on expert reuse frequency.
HOW THIS AFFECTS YOU
●
builderYou can run larger MoE models on consumer hardware with significantly higher token throughput.
●
researcherThe performance variance suggests expert reuse patterns are critical variables for MoE optimization effectiveness.