HyCo Hybrid Speculative Decoding Reduces Server GPU Load by 38.6%
October 16, 2026
HyCo optimizes mobile-cloud speculative decoding by utilizing a device-side NPU for both a small language model (SLM) and the server model's multi-token prediction (MTP) head. This approach reduces server GPU time per token by 38.6% on Spec-Bench and 29.2% during 32K-token summarization tasks.
HOW THIS AFFECTS YOU
●
builderYou can reduce inference costs and server-side latency for mobile applications by offloading drafting branches to the device NPU.