OoO-Spec Enables Parallel Semantic Speculation for Tool Calling
August 4, 2026
OoO-Spec uses a Qwen3-0.6B sidecar model to predict function choices and argument values in parallel with the main model's decoding. This out-of-order approach allows the target model to poll ready hints, reducing the total latency of tool-calling sequences.
HOW THIS AFFECTS YOU
●
builderYou can reduce tool-calling latency in production by implementing speculative sidecar architectures.
●
researcherYou can investigate the effectiveness of LoRA-trained sidecars for parallel semantic prediction.