Paged Memory Management for LLM Inference Serving

This direction is split into other directions: paged memory management. Its last version stays readable and citable here.

Version 1 of 139 papers50 conceptsfrozen 2026-10-04

Large Language Models (LLMs) are notoriously "stateful." To generate the next word in a sentence, the model must look back at every previous word. In the Transformer architecture, this history is stored as a Key-Value (KV) cache. As Shiyao Li and colleagues note in A Survey on Efficient Inference for Large Language Models, this cache grows linearly with the sequence length, often surpassing the size of the model weights themselves.

The challenge isn't just the size of the cache, but how we store it. Traditional systems allocate memory in large, contiguous chunks. This leads to massive waste—a problem known as memory fragmentation.

The Fragmentation Problem

When a system serves multiple users, it doesn't know how long each response will be. To be safe, traditional frameworks pre-allocate a contiguous block of GPU memory for the maximum possible sequence length.

As Xupeng Miao explains in Towards Efficient Generative Large Language Model Serving, this creates two types of waste:

  1. Internal Fragmentation: Memory reserved for tokens that are never generated.
  2. External Fragmentation: Small gaps of free memory between allocated blocks that are too small to be used for a new request.

This inefficiency acts as a "memory wall," limiting the number of concurrent requests (throughput) a single GPU can handle.

The Intuition: Virtual Memory for Tokens

The breakthrough solution, popularized by the vLLM library and analyzed by Yushuo Chen in Towards Coarse-to-Fine Evaluation of Inference Efficiency, is to treat GPU memory like an operating system treats RAM.

Instead of one giant contiguous block, the KV cache is broken into small, fixed-size pages (or blocks). These pages do not need to be next to each other in physical memory. A Block Table keeps track of the mapping: it tells the system that "Logical Page 1" of a user's conversation is actually stored in "Physical Block 7" on the GPU.

This "PagedAttention" mechanism allows for:

  • On-demand Allocation: Memory is only grabbed when a new token is actually generated.
  • Zero Internal Fragmentation: Since blocks are small and fixed-size, only the very last block of a sequence might have a tiny bit of unused space.
  • Flexible Sharing: If two requests share the same prefix (like a system prompt), they can both point to the same physical blocks, saving massive amounts of space.

Evolving the Paging Mechanism

While PagedAttention solved the fragmentation problem, it introduced a new hurdle: standard high-performance kernels (like FlashAttention) expect memory to be contiguous. To use paged memory, developers originally had to write complex, custom CUDA kernels.

Recent research has sought to bridge this gap. Ramya Prabhu and the team behind vAttention proposed using low-level CUDA Virtual Memory Management (VMM) APIs. This allows the system to map physical pages to a contiguous virtual address space. To the software kernel, the memory looks like one long string, but physically, it remains fragmented and efficient.

Similarly, Thomas Joshi demonstrated in Paged Attention Meets FlexAttention that these concepts can be integrated directly into PyTorch. By using a block table to resolve indices during the attention calculation, they achieved "near-zero memory waste" without leaving the standard Python-based ecosystem.

Scaling to Reasoning and the Edge

The importance of paged memory is magnifying as models become more complex. "Reasoning" models (like DeepSeek-R1) generate massive internal "chains of thought," putting extreme pressure on the KV cache. Akshat Ramachandran introduces ThinKV, which combines paged management with "thought-adaptive" compression to evict redundant tokens and reuse their memory slots instantly.

Even on smaller devices, the principles hold. Jeremy Stephen Gabriel Yee argues in On-Device LLMs for SMEs that paged memory is the primary mechanism that allows edge devices with limited RAM to support complex, long-context interactions.

By decoupling the logical sequence of a conversation from its physical location in VRAM, paged memory management has transformed LLM serving from a rigid, wasteful process into a dynamic, high-throughput system.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_paged_memory_management_for_llm_inferenc_4e132ebe,
  title = {Paged Memory Management for LLM Inference Serving},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/paged-memory-management-for-llm-inference-serving/rdr_4e132ebe56bd4f76943d42a6b011aa3e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_paged_memory_management_for_llm_inferenc_4e132ebe_v1,
  title = {Paged Memory Management for LLM Inference Serving},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/paged-memory-management-for-llm-inference-serving/rdr_4e132ebe56bd4f76943d42a6b011aa3e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}