Eigen Attention: Attention in Low-Rank Space for KV Cache Compression

Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, Kaushik Roy

Abstract

Large language models (LLMs) represent a groundbreaking advancement in the domain of natural language processing due to their impressive reasoning abilities. Recently, there has been considerable interest in increasing the context lengths for these models to enhance their applicability to complex tasks. However, at long context lengths and large batch sizes, the key-value (KV) cache, which stores the attention keys and values, emerges as the new bottleneck in memory usage during inference. To address this, we propose Eigen Attention, which performs the attention operation in a low-rank space, thereby reducing the KV cache memory overhead. Our proposed approach is orthogonal to existing KV cache compression techniques and can be used synergistically with them. Through extensive experiments over OPT, MPT, and Llama model families, we demonstrate that Eigen Attention results in up to 40% reduction in KV cache sizes and up to 60% reduction in attention operation latency with minimal drop in performance.

Introduction

As Large Language Models (LLMs) scale to handle longer context windows and higher batch sizes, the memory required for the Key-Value (KV) cache becomes a critical bottleneck. While model weights remain static, the KV cache grows linearly with sequence length and batch size. For instance, a Llama-2 7B model with a 32k context window and a batch size of 16 requires 256 GB for its KV cache, significantly exceeding the 14 GB needed for the model weights themselves. This memory-bound nature of inference limits throughput and increases hardware requirements. Eigen Attention addresses this by compressing the embedding dimension ($d$) of the KV cache, rather than just reducing token count or precision.

Methodology

Eigen Attention leverages the observation that attention inputs in LLMs are not full-rank, meaning they can be represented accurately using a smaller set of basis vectors. The process involves three main stages:

  1. Basis Generation: A small calibration dataset (e.g., a subset of WikiText-2) is used to collect query, key, and value activations during a forward pass.
  2. SVD Analysis: Singular Value Decomposition (SVD) identifies the principal components of these activations. The analysis reveals that a significant portion of the variance is captured by a small number of eigenvectors.
  3. Low-Rank Projection: The original weight matrices ($W^Q, W^K, W^V, W^O$) are projected into a lower-dimensional space ($r \ll d$). These projections are folded into the weights offline, allowing the model to naturally produce compressed keys and values during inference.
Comparison of (a) Standard Attention and (b) Eigen Attention architectures. The latter projects inputs into a low-rank space (r) before storage in the KV cache.

For models utilizing Rotary Positional Embeddings (RoPE), the authors implement a transformation mechanism to ensure the positional information remains compatible with the low-rank projection while maintaining the cache in its compressed form.

Empirical Analysis and Insights

A key finding of this work is that Key and Value matrices exhibit different levels of compressibility. SVD analysis shows that Key matrices typically have a lower rank than Value matrices, allowing for more aggressive compression of keys (up to 54% reduction) compared to values (around 26% reduction).

SVD analysis (a, b) showing cumulative eigenvalues for keys and values, and (c, d) the resulting automated layer-wise head dimension allotment.

Furthermore, the authors found that not all layers contribute equally to the model's performance. By using an automated "error budget" approach, Eigen Attention allocates different ranks to different layers, maximizing compression while maintaining accuracy. As seen in the analysis, larger models (e.g., Llama-2 70B) tend to be more resilient to this compression than smaller variants.

Results

Eigen Attention was evaluated across several model families, including OPT, MPT, and Llama, using benchmarks like WikiText-2, C4, and the lm-eval-harness.

  • Memory and Latency: The method achieved a consistent 40% reduction in KV cache size. In the OPT-66b model, this translated to a 60% reduction in attention operation latency.
  • Accuracy: At a 0.6x compression ratio (40% reduction), the drop in zero-shot accuracy across tasks like PiQA and HellaSwag was minimal (often within 1-3%). Performance could be further recovered using Low-Rank Adaptation (LoRA) fine-tuning.
  • Synergy: Eigen Attention is orthogonal to other methods. When combined with token pruning (e.g., H2O), it achieved a 0.24x total KV cache size with superior accuracy compared to pruning alone at 0.2x.

Conclusion and Future Work

Eigen Attention provides a robust, post-training solution for KV cache compression by exploiting the low-rank nature of attention embeddings. While it offers significant memory savings, future work will focus on optimizing the latency overhead introduced by RoPE transformations in certain architectures and integrating the method into high-throughput serving frameworks like vLLM.

Summary updated

Abstract

Large language models (LLMs) represent a groundbreaking advancement in the domain of natural language processing due to their impressive reasoning abilities. Recently, there has been considerable interest in increasing the context lengths for these models to enhance their applicability to complex tasks. However, at long context lengths and large batch sizes, the key-value (KV) cache, which stores the attention keys and values, emerges as the new bottleneck in memory usage during inference. To address this, we propose Eigen Attention, which performs the attention operation in a low-rank space, thereby reducing the KV cache memory overhead. Our proposed approach is orthogonal to existing KV cache compression techniques and can be used synergistically with them. Through extensive experiments over OPT, MPT, and Llama model families, we demonstrate that Eigen Attention results in up to 40% reduction in KV cache sizes and up to 60% reduction in attention operation latency with minimal drop in performance.
BibTeX
@inproceedings{saxena2024eigen,
  author = {Utkarsh Saxena and Gobinda Saha and Sakshi Choudhary and Kaushik Roy},
  title = {{Eigen Attention: Attention in Low-Rank Space for KV Cache Compression}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=BuAQnkzoA5},
}
Figure