Modern deep learning architectures have outgrown the physical memory limits of hardware accelerators. Whether dealing with the multi-gigabyte weight matrices of Mixture-of-Experts (MoE) models, the massive Key-Value (KV) caches required for million-token contexts, or the multi-terabyte embedding tables used in industrial recommendation systems, models routinely hit the "GPU memory wall."
To circumvent these constraints, systems researchers rely on memory offloading: keeping the bulk of a model's parameters or activations in high-capacity, low-cost storage tiers—such as host CPU RAM or Solid-State Drives (SSDs)—and streaming them to the GPU on demand.
However, this strategy introduces a severe performance penalty. While a GPU's internal High-Bandwidth Memory (HBM) operates at terabytes per second, the physical interconnect connecting the host to the accelerator (typically the PCIe bus) is orders of magnitude slower. This speed mismatch creates a massive interconnect bottleneck. As Yao et al. observe, the ability to effectively overlap data transfer with computation is highly sensitive to the ratio between interconnect bandwidth and local compute performance. When the data transfer time exceeds the execution time of a layer, the GPU starves—sitting completely idle while waiting for the next set of tensors to arrive.
To make offloading practical, recent research focuses on three core strategies to mitigate, hide, or entirely bypass this interconnect latency.
Exploiting Sparsity to Shrink the Transfer Budget
If the physical interconnect is a narrow pipe, the most direct way to avoid stalls is to send less data through it. Fortunately, deep neural networks exhibit significant structural and dynamic sparsity during inference, meaning only a fraction of the network is mathematically active for any given input.
Expert Sparsity in MoE Models
In Mixture-of-Experts architectures, each token triggers only a small subset of specialized layers (experts). Naive offloading systems stream all expert weights across the PCIe bus sequentially, which destroys throughput. To fix this, Kamahori et al. developed Fiddler, an orchestration framework that dynamically switches strategies based on batch size: it either schedules weight transfers to the GPU or temporarily routes data activations back to the CPU to compute the expert layers locally. To minimize transfers further, Zhong et al. introduce an adaptive gating mechanism that evaluates the "sensitivity" of experts, dynamically pruning inactive ones before they ever touch the hardware bus.
Activation Sparsity in Long-Context Attention
During long-context Large Language Model (LLM) generation, the KV cache grows linearly with sequence length, quickly exhausting GPU memory. When this cache is offloaded to host RAM, fetching it back for every token generation step creates an unsustainable I/O bottleneck.
To solve this, Yang 0011 et al. propose a "Double Sparsity" approach that combines token and channel sparsity to identify and fetch only the most critical feature channels offline. Taking this concept to the extreme, Synk demonstrates that by offloading the KV cache to a CPU-based vector database, a top-$k$ selection mechanism can retrieve less than 2% of the total context tokens for attention operations. By transferring only this tiny fraction of keys and values across the PCIe bus, the system achieves sublinear inference costs and unlocks million-token contexts on commodity hardware.
[Host Memory (CPU RAM / SSD)]
|
| -- (Pruned / Sparse Data Only) --
v
[PCIe Interconnect Bottleneck] <-- Reduced Data Volume = Lower Latency
|
v
[Accelerator (GPU HBM)]
Speculative Prefetching and Latency Hiding
When data transfers cannot be avoided, they must be hidden. This requires asynchronous execution: the system must predict what data will be needed in future computation steps and begin streaming it across the interconnect while the GPU is busy executing the current step.
The challenge lies in the non-deterministic nature of dynamic models. For instance, in LLM decoding, the exact attention weights or next-token paths cannot be known until the current token finishes computing. Waiting for this computation to finish before initiating a transfer guarantees a GPU stall.
To break this deadlock, researchers employ speculative prefetching. Jie et al. implement this concept for KV caches in SpeCache. The system maintains a low-precision, highly compressed "scout cache" directly on the GPU. Before a full attention kernel is launched, the scout cache speculatively predicts future attention patterns. The system uses these predictions to initiate host-to-device transfers of the full-precision KV pairs asynchronously. By the time the actual execution kernel requires the data, the tensors have already arrived in GPU memory, effectively hiding the interconnect latency.
Bypassing the Host and Near-Data Processing
The traditional data offloading path is inherently multi-step and inefficient: data is read from non-volatile storage (SSDs), copied into host CPU RAM, managed by the host operating system's file metadata structures, and finally pushed over the PCIe bus to the GPU. This pipeline introduces severe software and synchronization overheads.
Direct Memory Access
To streamline this path, Yuan et al. designed TERAIO, a framework that utilizes GPUDirect Storage (GDS). By establishing a direct Peer-to-Peer (P2P) DMA path between NVMe SSDs and GPU memory, the system bypasses the host CPU's main memory entirely. While the CPU still handles file system metadata operations, the heavy lifting of tensor movement is offloaded to dedicated hardware channels, allowing storage-based memory expansion to approach the speed of native training pipelines.
Near-Data Processing (NDP)
An even more radical paradigm shift eliminates the need to cross the interconnect altogether by moving the computation to where the data resides.
For recommendation models with massive embedding tables, Wilkening et al. propose performing "gather-reduce" operations directly on the SSD controller. Instead of flooding the PCIe bus with millions of small, random memory lookups, the SSD processes the lookups internally and transmits only the final, compact reduced vector to the host.
Similarly, for LLM inference, Liu et al. introduce Hermes, a system that offloads "cold" neural network weights to specialized memory modules called NDP-DIMMs. By executing sparse activations directly inside the system memory sticks, the framework transforms a slow data-transfer problem into a highly parallelized local computation problem, bypassing the PCIe bottleneck entirely.