A new PR in vLLM enables decoding of four FP8 e4m3 bytes per register on sm_89 architectures. This optimization improves throughput for FP8-quantized models on NVIDIA Ada Lovelace and Hopper GPUs.
HOW THIS AFFECTS YOU
●
builderYou can achieve higher inference throughput by leveraging optimized FP8 decoding on modern NVIDIA hardware.