Optimizer Gauge Symmetry Determines Low-Rank Recovery

The study shows that low-rank recovery in factored models depends on an optimizer's gauge equivariance. While gradient descent and Adam (with shared scalars) are equivariant, coordinate-wise methods like standard Adam and RMSProp are not.