Introduction
As Large Language Models (LLMs) are increasingly deployed for long-context applications—such as Retrieval-Augmented Generation (RAG), extensive document summarization, and codebase analysis—they encounter a significant computational hurdle: the Key-Value (KV) cache bottleneck. In autoregressive generation, LLMs store the KV vectors of previous tokens to avoid redundant computations. However, the memory footprint of this cache scales linearly with both sequence length and model depth. For instance, a 65B parameter model processing a 512K token context requires approximately 86GB of VRAM just for the KV cache, exceeding the capacity of a standard 80GB H100 GPU. This paper provides a systematic taxonomy and evaluation of techniques designed to compress the KV cache, reducing memory overhead and increasing inference throughput.
A Taxonomy of KV Cache Compression
Rather than viewing compression as a monolithic task, the authors categorize existing research into four distinct dimensions based on where the reduction occurs:
1. Compression Across Layers
These methods focus on reducing the redundancy found between different layers of the Transformer architecture. Techniques include Cross-Layer Attention (e.g., YOCO), where KV weights are shared across multiple layers, and Layer-Selective Attention, which involves dropping redundant attention layers entirely to streamline the model.
2. Compression Across Heads
Head-level compression reduces memory bandwidth requirements by sharing or grouping KV heads. Key strategies include:
- Multi-Query Attention (MQA): Using a single KV head for all query heads.
- Grouped-Query Attention (GQA): A middle ground that groups query heads to share a smaller number of KV heads.
- Multi-Head Latent Attention (MLA): Compressing KV states into a low-rank latent space to minimize storage.
3. Compression Across Tokens
Token-level strategies aim to reduce the effective sequence length. This is achieved through:
- State Space Models (SSMs): Architectures like Mamba replace the growing KV cache with a fixed-size recurrent state.
- Token Pruning: Dynamically removing tokens from the cache based on attention weights (e.g., LazyLLM, H2O).
- Prompt Compression: Using auxiliary models to filter out irrelevant words or sentences before they reach the primary LLM (e.g., LLMLingua, CPC).
4. Compression Across Hidden Dimensions
This dimension utilizes standard quantization techniques, such as converting high-precision (FP16) vectors into lower-precision formats (INT8 or INT4), thereby reducing the bits required per parameter in the stored vectors.
Implementation and Cost Paradigms
The paper further distinguishes these methods by their operational cost. Some architectures, such as YOCO or Mamba, require training from scratch, offering high efficiency at a high initial computational price. Others utilize post-training fine-tuning to retrofit existing models. Finally, training-free methods (like prompt compression) operate entirely during inference, making them highly accessible for existing deployments.
Key Insights and Empirical Results
The evaluation reveals several counter-intuitive findings regarding the relationship between compression and model performance:
- Compression as a Regularizer: Training-free methods like Context-aware Prompt Compression (CPC) and LongLLMLingua do not merely save memory; they can actually improve accuracy. By filtering out "noisy" or irrelevant context, these methods help the model focus on pertinent information. CPC achieved a 13.64% relative accuracy increase over a GPT-3.5-turbo baseline, while LongLLMLingua saw a 9.77% increase.
- Throughput and Latency: The efficiency gains are substantial. CPC delivered a 10.93x end-to-end speedup on A100 GPUs. Architectures trained from scratch also showed significant gains, with YOCO-3B achieving a 2.87x speedup in the prefilling stage on H100 hardware.
- The SSM-Transformer Hybrid: While pure SSMs like Mamba-2.8B show strong performance (6.21% improvement over baselines), hybrid models that combine SSM layers with standard attention are emerging as a necessary compromise to maintain high precision in tasks requiring exact in-context recall.
Limitations and Future Directions
The authors highlight a significant challenge in the current research landscape: the lack of standardized benchmarking. Because different papers use varying hardware (H100 vs. A100), batch sizes, and baseline models, direct "apple-to-apple" comparisons remain difficult. Future work must focus on establishing a unified framework to evaluate KV cache compression strategies under identical conditions to accurately map the Pareto frontier of accuracy, memory efficiency, and computational throughput.
Summary updated