BALANCE Framework for Hybrid LLM Inference in Edge Networks

BALANCE optimizes edge LLM deployment by dynamically assigning users to either autoregressive decoding or speculative decoding based on hardware constraints. The framework maximizes throughput by simultaneously running a large language model and a small language model to manage the latency-memory tradeoff.