Orthogonalization Drives Muon Optimizer Performance in Grokking

Ablation studies reveal that Muon's speedup over AdamW is driven by the Newton-Schulz orthogonalization process rather than spectral-norm constraints. Orthogonalizing optimizers reach generalization at approximately 3x lower spectral norm and settle into lower-norm solutions compared to standard methods.