Deep learning is currently hitting a "memory wall." While the computational power of GPUs has increased dramatically, the capacity of high-bandwidth memory (HBM) on these devices has not kept pace with the explosive growth of model parameters and context lengths. A trillion-parameter model or a million-token conversation simply cannot fit into the 40GB or 80GB of a single accelerator.
To bypass this physical limit, researchers have turned to hierarchical memory offloading. The core intuition is simple: the GPU should not be treated as a self-contained island, but as the top tier of a multi-level storage hierarchy that includes CPU RAM and NVMe SSDs. By treating GPU memory as a fast cache for active tensors and offloading inactive data to slower, high-capacity tiers, systems can handle workloads orders of magnitude larger than their physical VRAM would suggest.
The Hierarchy of Model State
In a typical training or inference workload, not all data is "active" at the same time. During training, optimizer states and gradients occupy the majority of memory but are only accessed at specific points in the iteration. Rajbhandari et al. demonstrated with ZeRO-Infinity that these states can be stored on NVMe storage and streamed to the GPU only when needed. This finding shifted the bottleneck from memory capacity to communication bandwidth, allowing a single GPU node to fine-tune models that previously required massive clusters.
For inference, the primary memory consumer is the Key-Value (KV) cache, which stores the history of a conversation to avoid redundant recomputation. As sequences grow to millions of tokens, the KV cache becomes a massive burden. Systems like Pensieve, proposed by Yu et al., transform LLM serving from "stateless" to "stateful" by migrating inactive conversation states between GPU and CPU memory. This allows a server to maintain thousands of concurrent long-form dialogues by using host RAM as a spill-over buffer.
Hiding Latency through Overlap
The fundamental challenge of offloading is the latency of the PCIe bus. Transferring data from CPU RAM to GPU VRAM is significantly slower than accessing VRAM directly. To make offloading viable, the system must hide this latency by overlapping data movement with computation.
The authors of RAGCache show that transferring intermediate model states over PCIe is still faster than recomputing those states from scratch (the "prefill" phase). By organizing cached document states in a "knowledge tree" across GPU and host memory, they reduce the time to first token (TTFT) by up to 4x. Similarly, in Mixture-of-Experts (MoE) models, where only a fraction of "expert" parameters are used for any given token, the authors suggest fetching required experts from host memory in parallel with the token dispatch process, ensuring the GPU never sits idle waiting for weights.
Selective Retrieval and Summarization
Rather than offloading data blindly, modern systems use "skimming" or "recall" mechanisms to decide what stays on the accelerator. This is often implemented as a tiered architecture:
- The Active Layer (GPU): Stores a compact global summary or a small subset of the most relevant tokens.
- The Passive Layer (CPU/NVMe): Stores the high-resolution, full-detail activations.
Qian et al. utilize this by keeping only a global summary on the GPU while offloading million-token details to the CPU. When the model identifies a specific relevant block via the summary, the system fetches the high-resolution data back to the GPU.
Other approaches, such as InfLLM by Xiao et al., use a Least Recently Used (LRU) eviction policy to manage this boundary. Synk et al. take this further by treating the CPU-offloaded KV cache as a searchable vector database. During inference, the GPU queries this database to retrieve only the <2% of tokens necessary for the current generation step, enabling million-token context processing on consumer-grade 16GB GPUs.
Memory as a Unified Abstraction
To manage these complex swaps, researchers are moving toward unified memory abstractions. Nie et al. introduced a fine-grained "Page" memory abstraction in Angel-PTM, which allows a unified scheduler to move data across GPU, CPU, and SSD tiers based on immediate demand.
This shift toward hierarchical management suggests that the "Memory Wall" is not a hard ceiling, but a transition point. By treating the GPU as a high-speed cache for a much larger, heterogeneous memory pool, the field is decoupling model scale from the physical constraints of silicon, enabling trillion-parameter intelligence on increasingly accessible hardware.