TL;DR
CompressKV is an efficient key-value (KV) cache compression framework that prevents critical prompt tokens from being accidentally evicted before generation begins. By identifying "Semantic Retrieval Heads"—attention heads that attend to entire semantic spans rather than just trivial prompt boundaries—and combining them with an offline, error-aware layer budget allocation, CompressKV maintains over 97% of full-cache question-answering performance using as little as 3% of the KV cache.
What Problem Does This Paper Address?
In large language models (LLMs), the KV cache stores previously computed keys and values to avoid redundant computation during text generation. For long inputs, however, the KV cache grows linearly, creating severe memory bottlenecks and slowing down inference.
To reduce cache size without retraining, existing methods evict "unimportant" tokens based on attention scores pooled across all attention heads within a group (especially under Grouped Query Attention, or GQA). The authors point out a major flaw in this approach: attention heads have functionally distinct behaviors. Specifically, streaming heads exclusively fixate on prompt boundaries (initial tokens and recent tokens). When pooled uniformly, these streaming heads dominate the score, causing models to discard crucial information in the middle of long texts.
Methods
CompressKV introduces two complementary mechanisms to compress the KV cache purely during the prefilling phase:
- Semantic Retrieval Head (SRH) Identification & Token Selection:
- Identifying SRHs: Conventional methods identify "retrieval heads" using rigid top-1 copy-and-paste rules on exact target tokens. CompressKV instead measures a head's attention aggregated across the entire answer span and its surrounding semantic context.
- Selection: During prefill, only the top-$k$ SRHs (defaulting to 4 per layer) vote on token importance within an observation window. Their pooled attention scores select a unified set of important token indices to keep, completely bypassing streaming-head interference.
- Offline Error-Aware, Layer-Adaptive Cache Allocation:
- Instead of spending runtime compute calculating attention entropy or variance dynamically, CompressKV profiles layer sensitivity offline.
- By simulating extreme KV compression (32 tokens per layer) and measuring the Frobenius norm of the difference in attention-block output compared to full cache, it assigns proportionally larger cache budgets to layers suffering higher reconstruction errors (bounded by minimum and maximum per-layer thresholds $[m, M]$).
[Observation Window] ──> [Filter Top-4 SRHs Only] ──> [Unified Token Eviction]
│
[Offline Layer Error Profiling] ───────────────> [Layer-Adaptive Budgeting]
Datasets
- LongBench: Evaluated across 16 tasks spanning single-document QA, multi-document QA, summarization, few-shot learning, synthetic tasks, and code completion. Evaluated on both LongBench and LongBench V2 (QA with chain-of-thought).
- Needle-in-a-Haystack (NIAH): Context lengths from 2K to 128K tokens to evaluate the retrieval of hidden target facts.
- Evaluated LLMs: Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct, with scaling verified on Qwen2.5-14B-Instruct and Qwen2.5-32B-Instruct.
Key Insights and Ideas
- Streaming heads drown out retrieval signals: In GQA, averaging attention across all heads in a group lets high-magnitude boundary attention from streaming heads wash out lower-magnitude but context-critical retrieval heads.
- Semantic heads matter far more than copy heads: Causal ablation reveals that masking the top 30 traditional retrieval heads in Mistral-7B causes only a ~12% accuracy drop, whereas masking the top 30 Semantic Retrieval Heads causes a staggering ~74% performance collapse.
- Layer importance is remarkably stable: The layer-wise budget allocation derived offline generalizes consistently across completely different tasks—deriving layer error from NarrativeQA alone yields nearly identical performance to averaging over the entire benchmark.
- Broad orthogonality: CompressKV is fully compatible with FlashAttention-2 and stacks cleanly with head-level budget allocation (HeadKV/AdaKV), prefilling accelerators (MInference/XAttention), and KV quantization (KIVI).
Key Results
- LongBench: CompressKV retains over 99% of full-cache accuracy with only a 19% KV cache budget, and preserves 97% of question-answering accuracy with just 3% KV cache. It outperforms strong baselines like SnapKV, PyramidKV, CAKE, and AdaKV, particularly under aggressive compression (e.g., budget size 256).
- Needle-in-a-Haystack: Achieves 90% accuracy on Llama-3.1-8B using only 0.7% of the full cache capacity (256 tokens) across up to 128K context lengths.
- Quantization Synergy: Combining CompressKV (pruning 87.5% of tokens) with KIVI 2-bit quantization yields a 98.44% memory reduction relative to 16-bit full cache while matching full-precision compression performance.
- Decoding Throughput: Delivers flat, near-constant decoding latency across context lengths from 4K up to 128K tokens on an NVIDIA A100 GPU.
Limitations and Future Work
- Offline Calibration Requirement: Identifying the Semantic Retrieval Heads and computing layer error scores requires an initial offline profiling pass on reference data before inference.
- Static Decoding Eviction: CompressKV performs eviction exclusively during the prefilling phase. In generation scenarios where output sequences are exceptionally long, secondary cache eviction during decoding may be needed.
- Fixed Per-Layer Head Count: The paper defaults to a fixed $k=4$ SRHs per layer; although adaptive-$k$ sweeps showed comparable results, dynamic per-layer head filtering could be explored further for diverse architectures.
Summary updated