EGGROLL Analyzes Low-Rank Evolution Strategies for LLM Optimization

EGGROLL replaces dense Gaussian weight perturbations with low-rank Gaussian products to make evolution strategies computationally feasible for LLMs. The research identifies that while these updates are exact on quadratic objectives, the finite-rank resolvent can introduce nonconservative components that may reverse local stability in smooth objectives.