As Large Language Models (LLMs) move toward processing millions of tokens, they encounter a physical bottleneck known as the "memory wall." The Key-Value (KV) cache—the memory that stores the intermediate states of a conversation to avoid recomputing them—grows linearly with sequence length. For a 100k-token context, this cache can easily exceed the 80GB capacity of a high-end NVIDIA H100 GPU.
To break this wall, researchers are moving away from treating GPU memory as the sole repository for model state. Instead, they are reimagining LLM serving as a hierarchical storage problem, where the KV cache is dynamically managed across a tiered architecture of GPU VRAM, system RAM, and persistent storage (SSDs).
The Core Insight: I/O is Faster Than Recomputation
The fundamental intuition behind hierarchical caching is a trade-off between computation and data transfer. When a user sends a prompt that overlaps with a previous conversation, the system has two choices: "prefill" the prompt by running it through the model again, or fetch the saved KV cache from memory.
As Chao Jin and colleagues demonstrate in RAGCache, transferring data over a PCIe bus from CPU RAM to the GPU is often significantly faster than the GPU time required to recompute thousands of tokens. This realization transforms host RAM from a passive component into a high-speed "spill-over" buffer. By treating the KV cache as a primary resource managed across a hierarchy, systems like Mooncake (developed by Mingxing Zhang and others) can handle contexts that are orders of magnitude larger than what fits in video memory.
Hiding the Latency of Slower Tiers
Moving data to CPU RAM or SSDs introduces latency. If the model has to wait for a cache to load from an SSD before it can generate the next token, the user experience suffers. Modern research focuses on "latency-hiding" techniques to make these transitions invisible.
One approach is Partial Recomputation. In KVPR, Chaoyi Jiang proposes a hybrid strategy: the GPU recomputes a small, critical portion of the KV cache while simultaneously streaming the bulk of the data from CPU memory. This overlaps computation with I/O, effectively masking the communication bottleneck.
Another strategy involves Granular Offloading. Rather than moving the entire cache for a request, HeadInfer by Cheng Luo and team offloads data at the level of individual attention heads. This fine-grained control allows the system to keep the most "active" heads in GPU memory while parking the rest in RAM, enabling a single consumer-grade GPU to process up to 4 million tokens.
Intelligent Eviction and Reuse
In a multi-tier system, deciding what to keep in the fastest tier (GPU VRAM) becomes a complex optimization problem. Standard policies like Least Recently Used (LRU) maximize hit rates but can ignore "tail latency"—the slow responses that frustrate users. Wenxin Zhang and Tianyi Peng introduced Tail-Optimized LRU (T-LRU) in Tail-Optimized Caching for LLM Inference to prioritize requests at risk of violating latency thresholds.
For structured programs where prompts share many common prefixes (like a multi-step agentic workflow), Lianmin Zheng and the SGLang team developed RadixAttention. This treats the KV cache as a tree structure (a radix tree), allowing different requests to share and reuse cached prefixes across the memory hierarchy. Similarly, TokenLake by Xuanzhe Liu and Xin Jin 0008 decouples this cache management from the model's scheduler, creating a unified "pool" of segments that can be elastically moved between tiers.
Compression as a Tiering Enabler
Finally, the efficiency of these hierarchies is amplified by compression. If the data being moved is smaller, the I/O bottleneck shrinks. Frameworks like Chelsea (Jie Hu et al.) use clustering to merge similar tokens, reducing the cache footprint by up to 80% before offloading. SnapStream by Nasim Farahini combines this with "sliding window" techniques to ensure that even as sequences grow to 100k+ tokens, the data being managed across the hierarchy remains lean and high-throughput.
By integrating these facets—hierarchical storage, latency-hiding I/O, and intelligent prefix sharing—researchers are transforming LLM serving from a memory-constrained bottleneck into a scalable, distributed systems problem.