Continuous Semantic Caching for Low-Cost LLM Serving

Semantic LLM response caching requires optimizing for an infinite query space rather than discrete sets. This framework uses dynamic epsilon-net discretization and Kernel Ridge Regression to approximate serving costs and arrival probabilities within continuous embedding spaces, bridging the gap between theoretical optimization and real-world semantic retrieval.