Pre-training Lexical Alignment Influences Zero-Shot Benchmark Accuracy

A study of word-level unigram cross-entropy (UCE) shows that zero-shot benchmark performance is partially driven by the lexical alignment between pre-training corpora and target tasks. Controlled from-scratch training experiments demonstrate how this lexical imprint scales with corpus identity.