As Large Language Models (LLMs) scale to handle increasingly long sequences, the memory footprint of the Key-Value (KV) cache becomes a critical bottleneck during inference. In standard autoregressive decoding, the model stores the keys and values of all past tokens to avoid recomputing them at every step. However, as highlighted by Li et al. in their comprehensive survey, A Survey on Large Language Model Acceleration based on KV Cache Management, this cache grows linearly with sequence length, rapidly exhausting GPU memory and limiting throughput.
To address this, a prominent research direction has emerged around dynamic KV cache eviction. Instead of retaining every historical token, these techniques dynamically evaluate and discard less significant tokens, allowing models to operate within a bounded memory budget while preserving long-range context.
The Foundation: Persistence of Importance
The intuition behind dynamic eviction relies on a core property of self-attention: not all tokens are created equal. A large portion of attention weight is concentrated on a small subset of critical tokens, such as initial tokens (attention sinks) or key structural and semantic anchors.
Desai et al. formalized this phenomenon as the "persistence of importance" hypothesis in Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time. They demonstrated that tokens that accumulate high attention scores early in the generation process tend to remain influential throughout the entire sequence. By tracking cumulative attention scores, systems can safely prune tokens that fall below a significance threshold.
Building on this, Lee introduced an exponential moving average of historical attention scores in Training Free Exponential Context Extension via Cascading KV Cache to ensure that consistently influential tokens are retained over long horizons. Similarly, Wu et al. utilized dot-product importance scoring alongside attention sinks in TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation to maintain a fixed-budget cache even when scaling up to 100,000 tokens.
Refining Importance: Beyond Raw Attention Scores
While early methods relied purely on raw attention weights, recent work demonstrates that attention scores alone do not fully capture a token's impact on the model's output.
For instance, Feng et al. analyzed cache management from an output perturbation perspective in Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective. The authors argued that effective cache management must account for the interaction between attention weights, value states, and output projection matrices. A token with a moderate attention score might still exert a massive influence on the final layer output if its projected value magnitude is exceptionally large.
Taking this sensitivity analysis further, An et al. proposed ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing. Instead of treating tokens in isolation, the authors modeled how token removal redistributes attention across the remaining context, using spatial-temporal smoothing to preserve the structural integrity of the hidden states.
Other approaches leverage contextual or architectural clues to determine importance. In ZigZagKV: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty, Zhong et al. utilized the final instruction tokens of a prompt to guide token selection, dynamically distributing the cache budget across layers based on layer uncertainty. Meanwhile, Bai et al. developed CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling, an inference-time technique that uses the user's specific instruction context to prevent the model from neglecting critical information during aggressive pruning.
Predictive and Learned Eviction
A fundamental limitation of heuristic eviction is that it is inherently backward-looking: it evaluates a token's past utility rather than its future relevance. To bridge this gap, researchers have introduced predictive and learned mechanisms.
Wang et al. addressed this in Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query by generating low-quality "pseudo-queries" to preview future token trajectories. This lookahead mechanism allows the system to prune the cache based on what the model is about to generate, rather than what it has already processed.
Alternatively, some frameworks replace hand-crafted heuristics with lightweight learned modules. In Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads, Yuan et al. introduced "retaining heads" trained to predict a token's future importance based solely on its current state, enabling efficient chunked processing. Similarly, Bui et al. integrated a learnable retention gate into attention heads in Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs. This gate predicts a scalar retention score at the moment of generation, allowing the model to filter out transient noise and retain long-term context.
System-Level Integration and Trade-offs
Dynamic eviction does not exist in a vacuum; it fundamentally changes how LLM serving systems manage memory hierarchies and multi-model pipelines.
For example, Chen et al. introduced ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction, which mitigates the risk of permanent information loss by backing up evicted tokens to CPU memory. It uses compact geometric "digests" to monitor real-time importance spikes, dynamically recalling offloaded pages back to the GPU when needed. In a different system context, Cha et al. developed SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences, which uses the attention weights of a large target model to dynamically prune the KV cache of a smaller draft model, ensuring alignment and high speculative acceptance rates.
Ultimately, as demonstrated by Yuan et al. in KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches, and corroborated by the reasoning benchmarks of Liu et al. in Hold Onto That Thought: Assessing KV Cache Compression On Reasoning, the choice of eviction strategy introduces distinct trade-offs. While heavy-hitter tracking preserves structural anchors for complex reasoning, overly aggressive compression can cause models to ramble or lose fine-grained details. The field continues to evolve by shifting from rigid, window-based discarding toward flexible, context-aware, and mathematically rigorous retention policies.
Key Foundational Work
Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads (2024) — contributed 2 concept elements to this direction.
Transformers are Multi-State RNNs (2024) — contributed 1 concept elements to this direction.
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches (2024) — contributed 1 concept elements to this direction.