Copy-Paste Language Models Improve Long-Context and Retrieval Performance
October 15, 2026
CPLMs add a lightweight Copy-Paste head to the standard language modeling head to improve in-context copying. Testing on models from 135M to 1.7B parameters shows improved training efficiency and downstream performance in long-context settings with negligible overhead.
HOW THIS AFFECTS YOU
●
builderImplementing a copy-paste head can improve retrieval and long-context accuracy for your specialized models.
●
researcherThe dual-head architecture offers an alternative gradient path that mitigates LM head bottlenecks.