vLLM has added support for MXFP8 linear weights to facilitate the inference of the DeepSeek V4 model. This allows for more efficient quantization and reduced memory overhead during model serving.
HOW THIS AFFECTS YOU
●
builderYou can achieve higher throughput and lower VRAM usage when serving DeepSeek V4 models.
●
researcherThis enables more efficient testing of quantized weights in large-scale inference environments.