Auxiliary Views Improve LLM Knowledge Acquisition During Pre-training

Allocating token budgets to auxiliary views—knowledge reformulations rather than document repetition—improves LLM learning and factual recall. The study shows these benefits occur regardless of the teacher model's strength and manifest through layer-wise compression mechanisms.