Flash-MSA Open-Source Kernels for Sparse Attention Training
July 12, 2026
Flash-MSA introduces performant open-source training kernels for Minimax Sparse Attention using CuTeDSL on Hopper and Blackwell architectures. The method utilizes blockwise sparsity with 128-token blocks and replaces MLA with GQA for the main attention mechanism to optimize training efficiency.
HOW THIS AFFECTS YOU
●
builderYou can now implement efficient sparse attention training on H100 and B200 GPUs.
●
researcherThis provides an open-source implementation of blockwise sparsity and GQA-based sparse attention for large-scale model training.