CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing

Yixuan Wang, Haoyu Qiao, Lujun Li, Sirui Han, Qingfu Zhu, Wanxiang CheSubmission to ICLR 2026 · Withdrawn

TL;DR

Existing training-free cross-layer KV cache sharing methods suffer steep performance drops because key-value representations across layers are surprisingly dissimilar. CommonKV solves this by factoring adjacent layers’ projection weights into shared and layer-specific matrices via offline Singular Value Decomposition (SVD), yielding a consistent latent KV cache that can be merged across layers without retraining. When combined with dynamic budget allocation, eviction, and quantization, CommonKV achieves up to a 98% KV cache compression ratio with minimal performance degradation.

What Problem Does This Paper Address?

In Large Language Models (LLMs), the Key-Value (KV) cache grows linearly with context length, creating a massive memory bottleneck during long-context generation. While sharing KV caches across Transformer layers is an appealing solution, existing approaches hit one of two walls:

  1. High training overhead: Architectural modifications (e.g., YOCO, MLKV) require pre-training from scratch, making them impractical for existing open-weight models.
  2. Quality collapse at high compression: Post-hoc, training-free merging methods (like MiniCache) degrade significantly when compression exceeds ~20%.

The root cause is that although adjacent layers' hidden states are highly similar (due to residual connections), their projected KV caches diverge substantially because each layer uses distinct projection weight matrices ($W_k, W_v$).

Methods

CommonKV bridges this gap through a training-free framework combining offline parameter decomposition with dynamic online cache merging:

  • Cross-Layer Concatenated SVD (Offline): For groups of adjacent layers (e.g., group size of 4), the authors concatenate their $W_k$ and $W_v$ matrices and perform truncated SVD. This decomposes the weights into a single shared projection matrix $A^{g_i}$ and compact layer-specific matrices $B_{k/v}^l$.
  • Consistent Latent KV Cache: Instead of storing raw key and value tensors, the model caches a latent state $h_i^l = x_i^l A^{g_i}$. Because adjacent input hidden states $x_i^l$ are already similar, multiplying them by the shared matrix $A^{g_i}$ yields latent representations that are vastly more consistent across layers.
  • Fisher Information-Weighted Merging: Merging within selected groups is computed as a weighted average of latent caches, weighted by the Fisher information of the corresponding layer weights (calculated offline using Wikitext-2 calibration data).
  • Adaptive Budget Allocation (Online): Because deeper layers exhibit higher inter-layer similarity than shallow layers, the model dynamically measures the cosine similarity of the latent cache across layers during inference and only merges groups above an adaptive similarity threshold.
  • Matrix Fusion for Fast Decoding: During decoding, to eliminate the computational cost of projecting latent states back to full values before multiplying by the output projection matrix $W_o$, the authors fuse $B_v^l$ with $W_o$ offline ($B_v^l W_o$).

Datasets

  • LongBench: A multitask bilingual benchmark evaluating real-world long-context tasks (QA, summarization, synthetic tasks, and code).
  • Ruler: A synthetic benchmark designed to evaluate long-context retrieval, aggregation, and multi-hop reasoning.
  • Wikitext-2: Used strictly offline (2,048 samples of length 1,024) to compute Fisher Information matrices for layer weighting.
  • Evaluated backbones: Llama3.1-8B-Instruct and Mistral-v0.2-7B-Instruct evaluated at context windows of 8k tokens.

Key Insights and Ideas

  • The KV Cache Mismatch Paradox: Residual connections ensure hidden states change smoothly across layers, but separate $W_k$ and $W_v$ projections scramble this alignment. Parameter sharing acts as an alignment mechanism, restoring the smooth transition in the latent cache space.
  • Overcoming GQA Dimensions: In Grouped-Query Attention (GQA), KV dimensions ($d_{kv}$) are smaller than hidden dimensions ($d_{hidden}$). SVD on individual KV matrices cannot compress below $d_{kv}$ without hurting capacity. Concatenating $W_k$ and $W_v$ across multiple layers increases the initial rank, making SVD effective for GQA models.
  • Orthogonality Across Compression Axes: Cross-layer sharing is fundamentally complementary to token eviction (e.g., SnapKV) and low-bit quantization (e.g., KVQuant). Merged latent caches can still be quantized to 4-bit and pruned across redundant token positions simultaneously.

Key Results

  • Higher Compression Ceilings: At a 50% (0.5) compression ratio, CommonKV maintains ~95% end-to-end task performance, achieving an average score of 71.59 on Llama3.1-8B (vs. 73.55 Full KV baseline), whereas Palu drops to 68.79 and ThinK collapses to 18.57.
  • Robustness at 60% Compression: CommonKV achieves 68.19 average score on Llama3.1-8B at a 0.6 compression ratio, outperforming Palu (50.59).
  • Minimal Runtime Overhead: Unlike online SVD methods (e.g., xKV, which spends >6× the prefilling latency on SVD), CommonKV’s dynamic similarity scoring adds only ~5% prefilling overhead, while offline matrix fusion keeps per-step decoding latency nearly identical to the uncompressed baseline.
  • Extreme 98% Compression: Cascading CommonKV (50% compression) + KVQuant (4-bit) + SnapKV token eviction achieves 98% KV cache footprint reduction with minor performance loss on LongBench.

Limitations and Future Work

  • Prefill-Only Compression: CommonKV compresses only the prompt (prefilling) tokens' KV cache to protect generation quality; generated tokens during decoding remain uncompressed (though they form a minor fraction of long-context workloads).
  • Position Re-computation: Because latent states lack Rotary Position Embeddings (RoPE), position embeddings must be applied dynamically during attention reconstruction.
  • Ultra-High Standalone Ratios: Like other training-free cross-layer methods, CommonKV experiences degradation when pushed past a standalone 60% compression ratio without pairing with quantization or eviction. Expanding cross-layer sharing across larger layer spans without quality loss remains an open challenge.

Summary updated

Abstract

Large Language Models (LLMs) confront significant memory challenges due to the escalating KV cache with increasing sequence length. As a crucial technique, existing cross-layer KV cache sharing methods either necessitate modified model architectures with subsequent pre-training or incur significant performance degradation at high compression rates. To mitigate these challenges, we propose CommonKV, a training-free method for cross-layer KV cache compression through adjacent parameters sharing. Inspired by the high similarity observed in cross-layer hidden states, we utilize Singular Value Decomposition (SVD) to achieve weight sharing across adjacent parameters, resulting in a more easily mergeable latent KV cache. Furthermore, we also introduce an adaptive budget allocation strategy. It dynamically assigns compression budgets based on cosine similarity, ensuring that dissimilar caches are not over-compressed. Experiments across multiple backbone models and benchmarks including LongBench and Ruler demonstrate that the proposed method consistently outperforms other low-rank and cross-layer approaches at various compression ratios. Moreover, we find that the benefits of CommonKV are orthogonal to other quantization and eviction methods. By integrating these approaches, we can ultimately achieve a 98% compression ratio without significant performance loss.
BibTeX
@misc{wang2025commonkv,
  author = {Yixuan Wang and Haoyu Qiao and Lujun Li and Sirui Han and Qingfu Zhu and Wanxiang Che},
  title = {{CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing}},
  year = {2025},
  eprint = {2508.16134},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.16134},
}

Loading related content...

Figure