This work formalizes KV cache eviction as a probabilistic expectation estimation problem rather than a heuristic task. The authors prove the problem is computationally hard and demonstrate that correcting for evicted entries during decoding improves performance.
HOW THIS AFFECTS YOU
●
builderYou can optimize inference throughput by applying more mathematically sound eviction and correction strategies.
●
researcherYou can move beyond heuristic-based cache management toward a formal probabilistic framework.