Multi-Objective Online Linear Programming for LLM Request Routing
July 7, 2026
A new routing framework uses online linear programming and bid-price control to manage LLM serving under KV-cache and batch-size constraints. The method employs warm-started projected first-order updates to maintain millisecond-level decision latency while optimizing latency-throughput trade-offs against explicit service-level objectives.
HOW THIS AFFECTS YOU
●
builderYou can better balance latency and throughput in production inference engines using SLO-aware routing.