TaskPress uses a task-guided meta-query during the prefill stage to prune the KV cache in a way that is reusable across different queries within the same task. It utilizes quantization scale factors as a zero-cost signal to identify and retain influential token outliers.
HOW THIS AFFECTS YOU
●
builderYou can significantly reduce long-context inference costs and latency by using task-specific, reusable cache compression.