Linking LLM Behavior to Training Data via Next-Token Distributions

LLM output distributions align closely with the empirical next-token distribution (ENTD) of their training data, with agreement scaling alongside model size and compute. Discrepancies between the two are concentrated in a specific long tail of input sequences related to architecture and finite-sample noise.