SGD Optimization Dynamics Modeled as a Percolation Process

Stochastic Gradient Descent (SGD) and Adam/AdamW exhibit variance spikes during training that correspond to architectural symmetries forcing subnetworks to merge in discrete blocks. This phenomenon is modeled as a percolation process where structural transitions create discrete scale invariance.