Differential Backpack Language Models Reduce Perplexity Gap by 31%
November 13, 2026
Differential Backpack models use signed contextualization weights to allow context to inhibit specific word senses. On WikiText-103, this method reduces the perplexity gap to Transformers by 31% and maintains performance up to 350M parameters on OpenWebText.
HOW THIS AFFECTS YOU
●
researcherThe use of differential attention for sense inhibition offers a more interpretable alternative to standard softmax-based models.