Introduction
Large Language Models (LLMs) are increasingly deployed in Retrieval-Augmented Generation (RAG) pipelines to handle long-context queries. However, the memory overhead of the Key-Value (KV) cache remains a significant bottleneck. While context caching allows for the reuse of precomputed document caches, multi-context scenarios introduce a unique challenge: cross-attention. When documents are cached independently, the model fails to capture inter-document relationships, leading to a loss in accuracy. Existing solutions either recompute the entire sequence—consuming massive resources—or ignore cross-attention entirely.
The SamKV Framework
To address these challenges, the authors propose SamKV, the first framework specifically designed for KV cache sparsification in multi-context environments. SamKV intelligently selects the most relevant 15% of the KV cache and performs selective recomputation to restore crucial cross-document relationships.
The framework consists of three core modules:
- Personalized Query Embedding Module: Instead of relying on a static user query, SamKV generates a "personalized" query for each document. By blending the user's query with consensus signals from other retrieved documents, the system identifies tokens that are globally relevant across the entire retrieved context.
- KV Selection Module: SamKV employs a dynamic Top-P sampling strategy rather than a fixed pruning ratio. It identifies "anchor points"—such as initial tokens and local context—and uses their attention patterns in the model's stable layers to adaptively determine which middle tokens to retain.
- Recomputation Module: To restore cross-attention, SamKV recomputes only the selected sparse tokens. It uses a "padding and merging" strategy to align important tokens across different layers, ensuring consistency. The system then integrates these new computations using either an Overwrite or Fusion strategy.
Experimental Results
The researchers evaluated SamKV using the LongBench benchmark (including datasets like HotpotQA and MuSiQue) across several models, including Mistral-7B-Instruct, Llama-3.1-8B-Instruct, and Qwen2.5-3B-Instruct.
Key Findings
- Efficiency and Compression: SamKV successfully reduces the KV cache to just 15% of its original size. This massive compression allows for higher throughput and lower GPU memory usage compared to methods like CacheBlend or EPIC.
- Accuracy Improvements: Surprisingly, SamKV often outperforms full-recomputation baselines. For instance, on Llama-3.1-8B (HotpotQA), SamKV achieved an F1 score of 35.27, compared to 24.12 for full recomputation. This suggests that sparsification acts as a noise filter, removing irrelevant information that might otherwise distract the model.
- Attention Stability: The study confirms that the final layers of LLMs exhibit stable attention patterns, which SamKV leverages to make more accurate pruning decisions.
Conclusion and Future Work
SamKV demonstrates that it is possible to achieve significant KV cache compression in multi-context scenarios without sacrificing accuracy. By focusing on inter-document consensus and stable attention layers, it provides a scalable solution for long-context RAG. Future research will focus on automating the identification of stable layers for different model architectures and exploring more advanced fusion weights for the recomputation step.
Summary updated