Linearity of Knowledge Predicts Generalization Capability After Fine-Tuning

A study across eleven models reveals that facts appearing linearly in the original pre-training data are more likely to support generalization during fine-tuning. The research distinguishes between active facts that support new relations and inert facts that only support direct recall.