SkewAdam Reduces MoE Training Memory by 97% via Tiered Allocation
July 20, 2026
SkewAdam optimizes Mixture-of-Experts memory by applying differentiated optimizer states to the backbone, experts, and router. For a 6.78B parameter model, it reduces optimizer state from 50.6 GB to 1.29 GB, cutting peak training memory from 81.4 GB to 31.3 GB.
HOW THIS AFFECTS YOU
●
builderYou can train larger MoE models on significantly less VRAM by utilizing tiered state allocation.
●
researcherThis provides a new method for scaling MoE architectures without proportional increases in memory overhead.