vLLM now skips fused FP8 qkv_proj tensors for model layers that do not utilize them. This optimization reduces unnecessary computations during the inference of MiMo-V2 models.
HOW THIS AFFECTS YOU
●
builderYou can achieve slight latency improvements when serving MiMo-V2 models through vLLM.