Overcoming the Linear Memory Growth of KV Caches

184 concepts169 papers1 proposal

Large Language Models (LLMs) have a massive appetite for memory, but the primary culprit during long-context inference isn't the model weights themselves—it is the Key-Value (KV) cache.

During autoregressive decoding, an LLM generates text token by token. To compute attention for a new token, the model must look back at all preceding tokens. Recomputing the Key and Value vectors for every historical token at every single step would be computationally disastrous. To avoid this, inference engines store these vectors in a dedicated memory buffer: the KV cache.

The fundamental issue is that the KV cache scales linearly with both the sequence length and the batch size. For a 7B parameter model processing a 128k context window, the KV cache can easily demand tens of gigabytes of VRAM per user request, quickly surpassing the memory required to store the model's parameters. This linear growth creates a hard physical wall, causing systems to run out of memory (OOM), forcing small batch sizes, and throttling throughput.

To unlock ultra-long context windows, a vibrant research direction has emerged to break this linear memory scaling. Rather than accepting the $O(N)$ memory tax, researchers are exploiting structural, mathematical, and semantic redundancies within LLMs to compress, evict, or entirely replace the KV cache.


Exploiting Redundancy via Low-Bit Quantization

The most direct way to shrink the KV cache footprint is to reduce the precision of the stored tensors. While model weights are routinely quantized to 4-bit or 8-bit integers, quantizing the KV cache is notoriously difficult. Keys and Values change dynamically with every prompt, and uniform quantization frequently triggers catastrophic accuracy degradation due to extreme outlier values across specific channels.

To make low-bit quantization work, researchers design methods that adapt to the unique geometric properties of activation distributions. For instance, Amir Gholami et al. developed KVQuant, which achieves near-lossless 3-bit compression. They observed that the distribution of keys changes significantly after applying positional rotations (like RoPE). By quantizing keys before this rotation and employing sensitivity-weighted non-uniform datatypes, they preserve the critical outlier dimensions that drive attention accuracy.

Similarly, Chengxi Liao et al. introduced Channel-Aware Mixed-Precision Quantization (ChanMix), which isolates "sensitive" channels that exhibit high variance and protects them with higher precision, while aggressively squeezing robust channels down to 2 bits.

Rather than treating every layer or token identically, modern quantization frameworks exploit the fact that the KV cache contains hidden linear dependencies. For example, the framework proposed by Alina Shutova et al., called AQUA-KV, leverages cross-layer correlations to predict cache entries, storing only the quantized residuals to achieve massive memory savings without sacrificing model quality.


Token Eviction and Sparsification

Not all words in a 100,000-token document are equally important at any given moment. Attention maps in deep Transformers are famously sparse; a query token typically concentrates its attention weight on a tiny fraction of historical tokens. Token eviction strategies exploit this sparsity by dropping or offloading historical KV pairs that are unlikely to be visited again.

The intuition behind modern eviction is that layers do not share the same contextual needs. As Chao Du et al. demonstrate with SimLayerKV, many layers act as "lazy layers" that primarily focus on the very first tokens (attention sinks) and the most recent tokens. By replacing full attention in these lazy layers with lightweight streaming attention, they drastically reduce the cache size.

Taking this layer-specific behavior further, Ziran Qin et al. introduced CAKE, which treats KV cache management as a dynamic allocation problem. Instead of enforcing a uniform token budget across the entire model, it adaptively shifts memory capacity to the layers that exhibit the highest "appetite" for long-range context, maintaining performance with a fraction of the original cache.

Standard KV Cache:  [Token 1] [Token 2] [Token 3] ... [Token N]  --> Grows infinitely
Eviction / Sinks:   [Sink 1]  [Sink 2]  ... [Recent t-1] [Recent t] --> Bounded memory

When tokens must be dropped to fit into GPU memory, permanent deletion can degrade reasoning if those tokens become relevant later. To solve this "permanent eviction" dilemma, Renze Chen et al. designed ArkVale, a system that offloads evicted KV pairs to host CPU memory and uses compact geometric "digests" to predict when a query might require recalling those tokens back to the GPU.


Structural and Functional Compression

Instead of dropping tokens or lowering their precision, another approach shrinks the underlying dimensions of the KV cache or condenses multiple tokens into unified representations.

Along the channel dimension, Luning Wang et al. introduced CSKV, which identifies structural redundancy across the hidden features of the keys and values. By projecting these vectors into a lower-dimensional subspace during inference, they achieve up to an 80% reduction in cache size with minimal training overhead.

Along the sequence dimension, context can be summarized. Qi Jia et al. proposed Context Compression with Sentinel Tokens, a method that forces the model to condense long spans of historical text into a single "sentinel" token. Instead of discarding information, the sentinel token acts as a lossy, semantic summary of the past context.

For scenarios where a single document is queried multiple times with different prompts, Jinuk Kim proposes KVzip. This query-agnostic compression technique identifies and retains only the core KV pairs necessary to mathematically reconstruct the original context, allowing a single compressed cache to be efficiently reused across distinct user queries.


Shifting to Constant-Memory Architectures

The most radical solution to the linear scaling problem is to abandon dense, full-history attention entirely. By shifting the underlying architecture toward recurrent or state-space formulations, the memory footprint required to maintain context can be compressed into a fixed-size state—achieving true $O(1)$ memory scaling relative to sequence length.

The RWKV architecture, detailed by Bo Peng et al., reformulates the Transformer's attention mechanism into a linear attention variant that can be executed like a Recurrent Neural Network (RNN). During inference, the model maintains a constant-sized hidden state rather than an ever-expanding cache of historical tokens.

Transformer Attention:  Query(t) × [Key(1), Key(2), ..., Key(t)]  --> O(N) Memory
Recurrent / State-Space: Input(t) + State(t-1) -> State(t)         --> O(1) Memory

To balance the high-quality in-context retrieval of Transformers with the hardware efficiency of RNNs, researchers have turned to hybrid architectures. Barak Lenz et al. introduced Jamba, a production-grade model that interleaves standard Transformer layers with Mamba State-Space Model (SSM) layers. Because the Mamba layers compress history into a fixed internal state, the hybrid model drastically reduces the frequency of KV cache generation, shrinking the overall memory footprint by up to 10x compared to pure attention models.

Alternatively, frozen pre-trained Transformers can be retrofitted with constant-memory plugins. Wonguk Cho et al. developed Modular Neural Memory (MoNe), which maps long-context inputs into layer-specific "fast-weight" neural memories. This allows the model to achieve constant query costs and massive VRAM savings while preserving the reasoning capabilities of the original frozen weights.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linear_kv_cache_scaling_3802fe69,
  title = {linear KV cache scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-kv-cache-scaling/rdr_3802fe6969b9483f843f72666a575668},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_kv_cache_scaling_3802fe69_v1,
  title = {linear KV cache scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-kv-cache-scaling/rdr_3802fe6969b9483f843f72666a575668/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake