Large Language Models (LLMs) face a fundamental "memory wall" during long-context generation. Because the Key-Value (KV) cache—the memory that stores previous token states—grows linearly with sequence length, models eventually run out of GPU memory or suffer from extreme latency. To solve this, a new consensus has emerged: rather than keeping every token, models should employ a hybrid retention strategy that combines a high-resolution local window with a sparse or compressed set of global "anchor" tokens.
The Stability of Attention Sinks
The most basic approach to reducing memory is a sliding window, where only the $N$ most recent tokens are kept. However, Xiao et al. et al. observed in Efficient Streaming Language Models with Attention Sinks that this causes catastrophic performance collapse. The reason is not just the loss of context, but a quirk of the Softmax operation: the model needs a "sink" to dump attention scores that don't fit elsewhere.
By preserving just the first few tokens of a sequence alongside the sliding window, the authors found that the model maintains structural stability. This "attention sink" mechanism allows models to process virtually infinite streams with constant memory overhead. Similarly, Ouyang et al. et al. in InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model and Chang et al. et al. in [Aha!
- Predicting What Matters Next: Online Highlight Detection Without Looking Ahead](/work/aha-predicting-what-matters-next-online-highlight-detection-xnef) extend this by "anchoring" task-specific instructions at the start of the cache, ensuring the model never forgets its primary objective even as the conversation history shifts.
Selective Sparsity and Importance Persistence
Beyond initial sinks, researchers have found that certain tokens in the distant past remain critical for future generation. Desai et al. et al. propose the "Persistence of Importance" hypothesis in Scissorhands, arguing that tokens which receive high attention early on will continue to be important. This allows for a hybrid cache that keeps a sliding window for local fluency and a sparse set of historically significant tokens for long-range dependencies.
Different methods have been proposed to identify these "important" tokens:
- Gumbel-based scoring: Adnan et al. et al. use a novel selection function in Keyformer to retain essential tokens.
- Information Hubs: Chen et al. et al. show in SepLLM that "meaningless" separator tokens like periods and commas act as natural hubs, and preserving them is sufficient to maintain global structure.
- Lookahead Probes: Wang et al. et al. in Lookahead Q-Cache generate "low-quality" preview tokens to predict which historical tokens the model will need next.
Tiered Compression and Multi-Resolution Memory
Instead of discarding tokens entirely, another branch of research focuses on tiered storage: keeping the immediate context in high resolution while compressing the distant past. Wang et al. et al. introduce CSKV, which stores historical tokens in a low-rank representation. Similarly, Zhang et al. et al. in KV Cache is 1 Bit Per Channel use a hybrid strategy where the sliding window remains in high precision while the rest of the cache is quantized to a single bit per channel.
This tiered approach is also applied to multimodal and agentic tasks. Yang and Zhao in StreamMem aggregate distant video frames into prototypes, while Ye et al. et al. in AgentFold "fold" previous agent actions into high-level summaries to maximize cache reuse.
Architectural Hybridization
Finally, some researchers argue that the Transformer architecture itself should be modified to support this hybrid view. Puvvada et al. et al. in SWAN-GPT and Yang et al. et al. in Rope to Nope and Back Again propose interleaving different types of attention layers. By using local sliding-window layers for most of the model and a few global layers for long-range retrieval, these architectures achieve massive context lengths with significantly reduced memory requirements.
In a different architectural shift, Shi et al. et al. propose LaCache, which distributes the cache across layers. In this "ladder-shaped" configuration, shallow layers focus on preserving early tokens while deeper layers prioritize the recent context, optimizing the total memory budget across the entire model stack.
Key Foundational Work
Efficient Streaming Language Models with Attention Sinks (2024) — contributed 1 concept elements to this direction.
Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference (2024) — contributed 1 concept elements to this direction.
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time (2023) — contributed 1 concept elements to this direction.