Unembedding Dynamics Reveal Capability Emergence Before Model Output

A study of Pythia's pretraining reveals that model hidden states encode task information before the readout matrix expresses it. Using parameter-trajectory crosscoding, researchers identified a window between 0.4% and 1.1% of pretraining where the readout reorganizes to enable WordNet category decoding.