The Graph Machine (GM) architecture uses differentiable edge-based routing to maintain O(n) state complexity. Replacing 75% of Qwen3-0.6B's dense layers with GM sparse layers during 15.7B token pretraining showed minimal loss degradation with only 2 of 4,096 tokens retrieved per KV head.
HOW THIS AFFECTS YOU
●
builderYou can achieve better efficiency in large-scale models by utilizing pointer-chasing mechanisms in sparse layers.
●
researcherThis offers a new pathway for scaling sparse models without the O(1) state restrictions of current methods.