Large Language Models (LLMs) are inherently stateful. During autoregressive generation, the model must remember the "keys" and "values" of every previous token to compute the next one. As Park et al. note, while the capabilities of these models are impressive, the "cold reality" of deployment is that this Key-Value (KV) cache creates a massive memory bottleneck.
Standard inference systems struggle with this because they typically allocate memory in large, contiguous blocks. This approach mirrors the way we might reserve a fixed-size parking lot for a car that hasn't arrived yet: we must reserve enough space for the longest possible sequence, even if the actual output is just a few words. As Chavan et al. and Wan et al. explain, this leads to significant internal fragmentation, where vast amounts of expensive GPU memory sit idle and unusable by other requests.
The Virtual Memory Analogy
To solve this, researchers have turned to a classic concept from operating systems: paging. Instead of forcing the KV cache to occupy a single contiguous stretch of memory, paged memory management treats GPU memory as a pool of non-contiguous "pages" or blocks.
In this paradigm, a sequence’s KV cache is broken into fixed-size segments. These segments can be scattered anywhere in the physical memory. A mapping table—similar to a virtual memory page table—keeps track of where each logical part of the sequence actually resides. Guo et al. and Li et al. highlight that this strategy effectively eliminates fragmentation. Because memory is allocated dynamically as the sequence grows, the system can utilize nearly 100% of its available VRAM.
Increasing Throughput and Efficiency
The primary benefit of paged attention is a dramatic increase in throughput. By reducing the memory footprint of each individual request, systems can process significantly more sequences in parallel.
- Higher Batch Sizes: Park et al. argue that non-contiguous allocation allows for much larger batch sizes within the same hardware constraints.
- Multi-Tenant Serving: In environments where multiple users share a GPU—such as serving various LoRA-adapted models—Chen et al. demonstrate that paged management allows the system to handle diverse requests with varying lengths without the overhead of static padding.
- Real-Time Interaction: For user-facing applications like those described by Hines et al., paging ensures low-latency token generation by optimizing how the cache is accessed during real-time interactions.
Beyond Simple Storage
The transition to paged memory has opened the door for more sophisticated cache manipulation techniques. Once the cache is organized into discrete, manageable pages, researchers can treat those pages as units of computation rather than just static data.
- Selective Loading: Tang et al. propose QUEST, which uses the paged structure to identify and load only the most relevant segments of the KV cache for a specific query. This allows for efficient long-context inference by avoiding the need to process the entire history for every token.
- State Streaming: Strati et al. (DéjàVu) explore decoupling prompt processing from generation by streaming KV cache pages between GPUs and CPUs, enabling the system to handle batches larger than the GPU memory itself.
- Parallel Generation: Cho et al. repurpose the cache infrastructure to partition input contexts across multiple units, accelerating the initial "prefill" phase of inference.
Challenges in Modern Architectures
While paging solves the fragmentation problem, it introduces new engineering complexities. Cai et al. note that integrating dynamic eviction—where redundant tokens are pruned to save space—with the structured nature of paged attention is a non-trivial task. Similarly, Jin et al. explore how to manage memory when models decode multiple tokens in parallel, requiring specialized attention masks to prevent interference within the shared memory pool.
Despite these challenges, paged memory management has become a foundational pillar of efficient LLM serving. By treating memory as a flexible resource rather than a rigid block, these systems overcome the "memory wall," enabling the deployment of larger models and longer contexts at a fraction of the previous cost.
Key Foundational Work
Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) — contributed 4 concept elements to this direction.
Inference Optimization of Foundation Models on AI Accelerators (2024) — contributed 3 concept elements to this direction.
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models (2024) — contributed 2 concept elements to this direction.