Output Embedding Centering for Stable LLM Pretraining

Output embedding centering (OEC) addresses output logit divergence by mitigating embedding anisotropy during LLM pretraining. Implementing OEC via deterministic mu-centering or mu-loss regularization stabilizes training more effectively than z-loss, matching the performance of logit soft-capping without merely masking the symptoms of geometric instability.