HyScale Speeds Long-Context Training by 2.07x via Selection Replay
October 26, 2026
HyScale optimizes learned sparse attention training for million-token contexts using HyTopK and selection replay. By avoiding full score table materialization, it achieves 2.07x faster GLM-5.1 training on 1M-token sequences compared to Megatron-LM on 512 H20 GPUs.
HOW THIS AFFECTS YOU
●
builderYou can reduce training costs and time when scaling models to million-token contexts.
●
researcherThis provides a more efficient system for training architectures like DeepSeek-V3.2.