Quadratic Model Accurately Predicts LLM Optimization Dynamics

A quadratic model can predict the optimization dynamics of a 150M parameter LLM trained on 3B tokens. By Taylor expanding the loss function at intermediate checkpoints, researchers can predict optimization behavior over windows lasting up to 10% of total training.