Multi-Dimensional Orthogonal KV Cache Compression

25 concepts21 papers2 proposals

Large Language Models (LLMs) are primarily constrained during inference not by their raw computational requirements, but by the memory footprint of the Key-Value (KV) cache. As context windows grow, this cache—which stores the attention states of all previous tokens—expands linearly, eventually exhausting GPU memory and limiting batch sizes.

While early optimization efforts focused on single-dimension solutions like quantization, a new consensus has emerged: the KV cache contains redundant information across multiple independent dimensions. By treating these dimensions as orthogonal, researchers have unlocked multiplicative memory savings that far exceed the limits of any single technique.

The Four Dimensions of Redundancy

To understand why orthogonal compression works, we must view the KV cache as a multi-dimensional tensor. Redundancy exists in four distinct axes:

  1. The Token Dimension (Sequence Length): Not all tokens are equally important for future predictions. Desai et al. in Scissorhands demonstrate the "Persistence of Importance" hypothesis, showing that a small set of influential tokens can be retained while others are evicted. Similarly, Nawrot et al. propose Dynamic Memory Compression, which learns to merge redundant tokens rather than simply deleting them.
  2. The Channel Dimension (Hidden States): Within a single token's representation, many feature channels are redundant or static. Xu introduces ThinK, which uses query-driven pruning to identify and remove these redundant channels. The authors of LeanK further this by using a two-stage distillation process to prune unimportant channels in the Key cache specifically.
  3. The Layer Dimension (Depth): Hidden states often remain highly similar across adjacent layers. Brandon et al. propose Cross-Layer Attention (CLA), which shares KV activations vertically across layers. Wu et al. extend this in LCKV, caching pairs for only a subset of layers to achieve massive throughput gains.
  4. The Precision Dimension (Numerical): Standard 16-bit representations are often overkill. Quantization reduces the bit-width of each stored value, a technique that Jiang et al. show in Back Razor is fundamentally compatible with sparsification.

The Power of Orthogonality

The core intuition behind this research direction is that these dimensions are functionally independent. If you reduce the number of tokens by 50% and then reduce the number of channels per token by 50%, you haven't just saved 50% of the memory—you've saved 75%.

This multiplicative effect is the "holy grail" of cache optimization. Chang et al. demonstrate this synergy in Palu, where low-rank projection of the hidden dimension is combined with quantization to achieve speedups that neither could reach alone. Similarly, Wang et al. show in CSKV that channel shrinking and quantization together can slash the cache footprint by up to 95%.

Low-Rank Latent Spaces

A recurring mechanism for achieving this multi-dimensional squeeze is the projection of KV states into a low-rank latent space. Instead of storing the full-width Key and Value vectors, the model stores a compressed "bottleneck" representation.

Wang et al. highlight this in the DeepSeek architecture, where Multi-head Latent Attention (MLA) uses low-rank joint compression to minimize the KV cache during inference. Saxena et al. take a post-training approach with Eigen Attention, identifying the most important principal components of the data to project the cache into a lower-rank space without retraining.

Synergistic Architectures

The most advanced systems now stack these dimensions intentionally. Wang et al. demonstrate in CommonKV that by integrating cross-layer sharing, token eviction, and quantization, it is possible to reach a staggering 98% reduction in cache size.

This shift suggests that the future of efficient LLMs lies not in finding a better pruning algorithm, but in designing architectures where every dimension of the attention tensor is compressed. As Shi et al. argue in KV-Latent, combining dimensional-level reduction with head-reduction and token-eviction creates a robust framework for long-context inference that remains performant even under extreme memory constraints.

Key Foundational Work

Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time (2023) — contributed 1 concept elements to this direction.
Back Razor: Memory-Efficient Transfer Learning by Self-Sparsified Backpropagation (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_orthogonal_cache_compression_9175e630,
  title = {orthogonal cache compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/orthogonal-cache-compression/rdr_9175e630784d4140bd5ad4108c8e349c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_orthogonal_cache_compression_9175e630_v1,
  title = {orthogonal cache compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/orthogonal-cache-compression/rdr_9175e630784d4140bd5ad4108c8e349c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake