TL;DR
Existing training-free cross-layer KV cache sharing methods suffer steep performance drops because key-value representations across layers are surprisingly dissimilar. CommonKV solves this by factoring adjacent layers’ projection weights into shared and layer-specific matrices via offline Singular Value Decomposition (SVD), yielding a consistent latent KV cache that can be merged across layers without retraining. When combined with dynamic budget allocation, eviction, and quantization, CommonKV achieves up to a 98% KV cache compression ratio with minimal performance degradation.
What Problem Does This Paper Address?
In Large Language Models (LLMs), the Key-Value (KV) cache grows linearly with context length, creating a massive memory bottleneck during long-context generation. While sharing KV caches across Transformer layers is an appealing solution, existing approaches hit one of two walls:
- High training overhead: Architectural modifications (e.g., YOCO, MLKV) require pre-training from scratch, making them impractical for existing open-weight models.
- Quality collapse at high compression: Post-hoc, training-free merging methods (like MiniCache) degrade significantly when compression exceeds ~20%.
The root cause is that although adjacent layers' hidden states are highly similar (due to residual connections), their projected KV caches diverge substantially because each layer uses distinct projection weight matrices ($W_k, W_v$).
Methods
CommonKV bridges this gap through a training-free framework combining offline parameter decomposition with dynamic online cache merging:
- Cross-Layer Concatenated SVD (Offline): For groups of adjacent layers (e.g., group size of 4), the authors concatenate their $W_k$ and $W_v$ matrices and perform truncated SVD. This decomposes the weights into a single shared projection matrix $A^{g_i}$ and compact layer-specific matrices $B_{k/v}^l$.
- Consistent Latent KV Cache: Instead of storing raw key and value tensors, the model caches a latent state $h_i^l = x_i^l A^{g_i}$. Because adjacent input hidden states $x_i^l$ are already similar, multiplying them by the shared matrix $A^{g_i}$ yields latent representations that are vastly more consistent across layers.
- Fisher Information-Weighted Merging: Merging within selected groups is computed as a weighted average of latent caches, weighted by the Fisher information of the corresponding layer weights (calculated offline using Wikitext-2 calibration data).
- Adaptive Budget Allocation (Online): Because deeper layers exhibit higher inter-layer similarity than shallow layers, the model dynamically measures the cosine similarity of the latent cache across layers during inference and only merges groups above an adaptive similarity threshold.
- Matrix Fusion for Fast Decoding: During decoding, to eliminate the computational cost of projecting latent states back to full values before multiplying by the output projection matrix $W_o$, the authors fuse $B_v^l$ with $W_o$ offline ($B_v^l W_o$).
Datasets
- LongBench: A multitask bilingual benchmark evaluating real-world long-context tasks (QA, summarization, synthetic tasks, and code).
- Ruler: A synthetic benchmark designed to evaluate long-context retrieval, aggregation, and multi-hop reasoning.
- Wikitext-2: Used strictly offline (2,048 samples of length 1,024) to compute Fisher Information matrices for layer weighting.
- Evaluated backbones: Llama3.1-8B-Instruct and Mistral-v0.2-7B-Instruct evaluated at context windows of 8k tokens.
Key Insights and Ideas
- The KV Cache Mismatch Paradox: Residual connections ensure hidden states change smoothly across layers, but separate $W_k$ and $W_v$ projections scramble this alignment. Parameter sharing acts as an alignment mechanism, restoring the smooth transition in the latent cache space.
- Overcoming GQA Dimensions: In Grouped-Query Attention (GQA), KV dimensions ($d_{kv}$) are smaller than hidden dimensions ($d_{hidden}$). SVD on individual KV matrices cannot compress below $d_{kv}$ without hurting capacity. Concatenating $W_k$ and $W_v$ across multiple layers increases the initial rank, making SVD effective for GQA models.
- Orthogonality Across Compression Axes: Cross-layer sharing is fundamentally complementary to token eviction (e.g., SnapKV) and low-bit quantization (e.g., KVQuant). Merged latent caches can still be quantized to 4-bit and pruned across redundant token positions simultaneously.
Key Results
- Higher Compression Ceilings: At a 50% (0.5) compression ratio, CommonKV maintains ~95% end-to-end task performance, achieving an average score of 71.59 on Llama3.1-8B (vs. 73.55 Full KV baseline), whereas Palu drops to 68.79 and ThinK collapses to 18.57.
- Robustness at 60% Compression: CommonKV achieves 68.19 average score on Llama3.1-8B at a 0.6 compression ratio, outperforming Palu (50.59).
- Minimal Runtime Overhead: Unlike online SVD methods (e.g., xKV, which spends >6× the prefilling latency on SVD), CommonKV’s dynamic similarity scoring adds only ~5% prefilling overhead, while offline matrix fusion keeps per-step decoding latency nearly identical to the uncompressed baseline.
- Extreme 98% Compression: Cascading CommonKV (50% compression) + KVQuant (4-bit) + SnapKV token eviction achieves 98% KV cache footprint reduction with minor performance loss on LongBench.
Limitations and Future Work
- Prefill-Only Compression: CommonKV compresses only the prompt (prefilling) tokens' KV cache to protect generation quality; generated tokens during decoding remain uncompressed (though they form a minor fraction of long-context workloads).
- Position Re-computation: Because latent states lack Rotary Position Embeddings (RoPE), position embeddings must be applied dynamically during attention reconstruction.
- Ultra-High Standalone Ratios: Like other training-free cross-layer methods, CommonKV experiences degradation when pushed past a standalone 60% compression ratio without pairing with quantization or eviction. Expanding cross-layer sharing across larger layer spans without quality loss remains an open challenge.
Summary updated