Layerwise Repacking Reduces Communication Overhead in Hybrid-Attention Training

Layerwise Repacking optimizes variable-length training for hybrid-attention models by repacking samples into smaller, node-local sequence-parallel groups at sparse-attention boundaries. This approach prevents the need for extra redistribution passes while maintaining high-performance wide groups for full-attention layers.