Pruning Redundant Attention Heads to Suppress Verbatim Memorisation
November 2, 2026
Zeroing out a substantial fraction of redundant attention heads during inference significantly reduces the verbatim reproduction of training data. This method suppresses memorization-driven privacy risks while maintaining or improving general language modeling performance on standard benchmarks.
HOW THIS AFFECTS YOU
●
researcherYou can leverage head pruning to decouple associative memory from general reasoning capabilities.
●
policyThis offers a potential technical mechanism to mitigate privacy leakage and IP exposure in LLMs.