CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing

Yixuan Wang, Haoyu Qiao, Lujun Li, Qingfu Zhu, Wanxiang Che

Abstract

Large Language Models (LLMs) confront significant memory challenges due to the escalating KV cache with increasing sequence length. As a crucial technique, existing cross-layer KV cache sharing methods either necessitate modified model architectures with subsequent pre-training or incur significant performance degradation at high compression rates. To mitigate these challenges, we propose CommonKV, a training-free method for cross-layer KV cache compression through adjacent parameters sharing. Inspired by the high similarity observed in cross-layer hidden states, we utilize Singular Value Decomposition (SVD) to achieve weight sharing across adjacent parameters, resulting in a more easily mergeable latent KV cache. Furthermore, we also introduce an adaptive budget allocation strategy. It dynamically assigns compression budgets based on cosine similarity, ensuring that dissimilar caches are not over-compressed. Experiments across multiple backbone models and benchmarks including LongBench and Ruler demonstrate that the proposed method consistently outperforms existing low-rank and cross-layer approaches at various compression ratios. Moreover, we find that the benefits of CommonKV are orthogonal to other quantization and eviction methods. By integrating these approaches, we can ultimately achieve a 98% compression ratio without significant performance loss.

The Memory Wall in Long-Context LLMs

As Large Language Models (LLMs) scale to handle context windows exceeding 100k tokens, the KV Cache becomes a primary bottleneck. Storing the Keys and Values for every token across every layer leads to a memory footprint that grows linearly with sequence length. In many long-context scenarios, this cache can dwarf the model's weights, making deployment on standard hardware prohibitively expensive.

While architectural changes like Multi-Query Attention (MQA) address this, they require retraining. Post-hoc methods like quantization or token eviction are popular but often reach a performance ceiling. CommonKV addresses a specific gap: making cross-layer sharing—where multiple layers share the same cache—viable without retraining.

Methodology: The Hidden State Paradox

The core innovation of CommonKV stems from a key observation: while the final Key and Value tensors in adjacent layers are mathematically distinct, the hidden states entering those layers are remarkably similar due to residual connections.

Cosine similarity of various components across adjacent layers, highlighting the high similarity of hidden states compared to raw KV caches.

To exploit this, CommonKV introduces a training-free pipeline:

  1. Cross-layer Parameter Sharing via SVD: Weight matrices ($W_k, W_v$) are decomposed using Singular Value Decomposition into a shared component ($A$) and layer-specific components ($B$).
  2. The Latent KV Cache: Instead of caching final KV vectors, the model caches a "latent" representation ($h = xA$). Because $A$ is shared and the input $x$ is similar across layers, this latent cache is highly consistent and can be merged across layers with minimal information loss.
  3. Adaptive Budget Allocation: The method dynamically assigns compression budgets. Layers with lower similarity receive more bits/tokens, while highly similar layers are compressed more aggressively.
  4. Fisher Information-based Merging: When combining caches, Fisher Information weights the importance of parameters, ensuring the most "informative" data is preserved.

Key Results

CommonKV was evaluated on LongBench and Ruler using Llama 3.1-8B and Mistral-v0.2-7B backbones. The results demonstrate a significant leap over previous state-of-the-art methods like MiniCache.

Average score across various compression ratios, showing CommonKV's superior performance retention at high compression levels.

On Llama 3.1-8B, CommonKV maintains over 95% of original performance at a 50% compression ratio. Furthermore, the method is orthogonal to other techniques. By stacking CommonKV with 4-bit quantization (KVQuant) and token eviction (SnapKV), the researchers achieved a 98% total compression ratio with negligible accuracy loss.

Conclusion

CommonKV provides a "plug-and-play" solution for the LLM memory wall. By shifting the sharing mechanism to a latent space and utilizing offline SVD, it achieves high compression efficiency without the need for expensive retraining. Its training-free nature and compatibility with existing quantization methods make it a highly practical tool for deploying long-context LLMs on constrained hardware.

Summary updated

Abstract

Large Language Models (LLMs) confront significant memory challenges due to the escalating KV cache with increasing sequence length. As a crucial technique, existing cross-layer KV cache sharing methods either necessitate modified model architectures with subsequent pre-training or incur significant performance degradation at high compression rates. To mitigate these challenges, we propose CommonKV, a training-free method for cross-layer KV cache compression through adjacent parameters sharing. Inspired by the high similarity observed in cross-layer hidden states, we utilize Singular Value Decomposition (SVD) to achieve weight sharing across adjacent parameters, resulting in a more easily mergeable latent KV cache. Furthermore, we also introduce an adaptive budget allocation strategy. It dynamically assigns compression budgets based on cosine similarity, ensuring that dissimilar caches are not over-compressed. Experiments across multiple backbone models and benchmarks including LongBench and Ruler demonstrate that the proposed method consistently outperforms existing low-rank and cross-layer approaches at various compression ratios. Moreover, we find that the benefits of CommonKV are orthogonal to other quantization and eviction methods. By integrating these approaches, we can ultimately achieve a 98\% compression ratio without significant performance loss.
BibTeX
@misc{wang2025commonkv,
  author = {Yixuan Wang and Haoyu Qiao and Lujun Li and Qingfu Zhu and Wanxiang Che},
  title = {{CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing}},
  year = {2025},
  eprint = {2508.16134},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.16134},
}
Figure