Structural priors from symbolic data improve language modeling loss

Training models on non-language data like music and cellular automata can induce useful structural priors for natural language. While these methods reduce next-token-prediction loss and minimize weight shifts during language training, they do not consistently improve downstream linguistic performance.