The vLLM project has implemented a fix for MiMo-V2.5 fused FP8 qkv_proj sharding. This update addresses issues with the pre-shard count and the Multi-Token Prediction (MTP) path.
HOW THIS AFFECTS YOU
●
builderYou can benefit from improved FP8 sharding performance and stability in the vLLM inference runtime.
●
researcherThis update improves the support for specific FP8 quantization schemes in high-performance inference.