Modifying training distributions to align atomic facts with the positional structure of retrieval circuits enables GPT-2-like models to generalize compositional reasoning out-of-distribution. This method requires no architecture changes or additional supervision.
HOW THIS AFFECTS YOU
●
researcherYou can improve OOD compositional generalization by carefully structuring the positional alignment of facts during training.