Co-adaptation Fine-tuning for Long-Context Sparse Attention
August 21, 2026
A new fine-tuning method allows transformer models to co-adapt with any KV cache sparse attention policy using moderate hardware like a single A100. The KeysAndValues open-source library provides an efficient implementation of H2O sparse attention with dedicated scaled dot-product attention kernels.
HOW THIS AFFECTS YOU
●
builderYou can use the KeysAndValues library to reduce long-context inference costs on single-GPU setups.
●
researcherThis method demonstrates that co-adapting with sparse policies can outperform exact attention models.