Adaptive Token Eviction and KV Cache Compression

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 21,077 papers1,095 conceptsfrozen 2026-10-04

As Large Language Models (LLMs) scale to handle longer contexts, the Key-Value (KV) cache—the memory used to store past activations—has become the primary bottleneck for inference. In long-sequence tasks, the KV cache can easily exceed the size of the model weights themselves, leading to massive memory consumption and slowed throughput.

The emerging research direction in this field moves away from "one-size-fits-all" pruning toward adaptive, multi-axis compression. Instead of treating all tokens or attention heads equally, researchers are developing frameworks that allow the model to "diagnose" its own internal state and dynamically discard or compress information that is redundant for the current task.

The Diagnose-Before-Compress Paradigm

The core intuition behind adaptive compression is that attention is not uniform. In a typical Transformer, different attention heads serve distinct roles: some focus on the immediate local context, while others track "special tokens" (like the beginning-of-sentence token) or global semantic markers.

Suyu Ge and Jiawei Han introduced FastGen (Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs), which formalizes this as a "diagnose-before-compress" framework. By profiling attention heads during the initial prompt phase, FastGen identifies which heads are "local" and which are "global." It then applies tailored eviction policies to each head—for instance, keeping only a small sliding window for local heads while retaining a broader set of tokens for global ones. This allows for a 40-50% reduction in memory without retraining the model.

Balancing Global and Local Importance

A significant challenge in token eviction is "context bias"—the tendency to over-rely on recent tokens while losing the "thread" of the global conversation. To counter this, Ye Li and colleagues proposed EMS (Adaptive Evict-then-Merge Strategy).

EMS works by calculating a "Global-Local" importance score. Instead of just deleting tokens, it uses an Evict-then-Merge strategy: irrelevant data is discarded, but redundant information is merged together. This ensures that the "signal" from older tokens is preserved in a compressed form rather than being silenced entirely. Similar logic is applied in Vision Transformers (ViTs) by Jung Hwan Heo in Delayed Spatial Merging (DSM) (Training-Free Acceleration of ViTs with Delayed Spatial Merging), which delays token merging until later layers where tokens have already diversified, preventing early information loss.

Multi-Modal and Speculative Pruning

The need for adaptive pruning is even more acute in multi-modal models, where video or audio tokens can quickly overwhelm the system.

The Synergy Frontier: Sparsity meets Precision

The most pressing research question across these papers is the interaction between orthogonal compression axes: structural pruning (dropping tokens) and numerical quantization (reducing bit-precision).

Historically, these techniques were developed in isolation. However, Akshat Sharma and Abhishek Sharma demonstrated in MiniKV (Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache) that these methods can be synergistic. By co-designing 2-bit quantization with adaptive eviction, they achieved 80% cache compression.

The difficulty lies in the fact that pruning changes the underlying data distribution. As noted in Focused Quantization (Focused Quantization for Sparse CNNs), pruning creates "holey" weight distributions that can invalidate standard quantization assumptions. Research like Harmonious Coexistence (Harmonious Coexistence of Structured Weight Pruning and Ternarization) suggests that without careful "stabilization," combining these methods leads to compounding errors.

Why It Works: Exploiting Redundancy

The success of these methods rests on a fundamental observation about Transformers: Attention Collapse. As models get deeper, attention maps often become nearly identical across layers (DeepViT by Daquan Zhou). Adaptive eviction works because it identifies this redundancy in real-time. By treating the model as a "black box" that can signal its own priorities—whether through attention scores, audio cues, or layer-wise profiling—researchers are turning the Transformer's inherent over-parameterization into an opportunity for extreme efficiency.

Papers in this version

Showing 60 of 1,077 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 785 papers
  2. Version 1 · 2026-10-04 · 1077 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_orthogonal_compression_integration_99d3aead,
  title = {orthogonal compression integration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/orthogonal-compression-integration/rdr_99d3aead579b4ec98ed3bf2d8044f2fe},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_adaptive_token_eviction_and_kv_cache_com_99d3aead_v1,
  title = {Adaptive Token Eviction and KV Cache Compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adaptive-token-eviction-and-kv-cache-compression/rdr_99d3aead579b4ec98ed3bf2d8044f2fe/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}