Copy-Paste Language Models improve in-context copying efficiency
October 12, 2026
CPLMs add a lightweight Copy-Paste head to standard language modeling heads to improve in-context copying capabilities. Tested on models from 135M to 1.7B parameters, the method improves training efficiency and performance in long-context and retrieval settings with minimal overhead.
HOW THIS AFFECTS YOU
●
builderYou can implement this lightweight head to improve performance in RAG or long-context applications.
●
researcherThis provides a new architectural approach to mitigating the LM head gradient bottleneck.