Reevaluating LLM training beyond simple next-token prediction

Autoregressive training loops focus on increasing the probability of the actual next token, but this view is a zeroth-order approximation of the complex gradient-based parameter updates occurring under the hood. Treating models solely as next-token predictors ignores the internal mechanisms required to model complex distributions.