vLLM supports speculative decoding on AMD hardware, utilizing a draft-and-verify mechanism to commit multiple tokens per target-model pass. Throughput improvements depend on the draft model's acceptance rate, proposal length, and specific model architecture.
HOW THIS AFFECTS YOU
●
builderYou can improve inference throughput on AMD GPUs by implementing draft-and-verify workflows.