Deep learning performance has long been tied to the scale of neural network architectures. However, as models grow exponentially into billions or trillions of parameters, they hit a physical barrier: the accelerator memory wall. While computational throughput (FLOPs) scales rapidly, the high-bandwidth on-device memory (VRAM) of individual graphics processing units (GPUs) scales linearly and much more slowly.
To understand why this is a bottleneck, consider what must fit into a GPU's memory during training. It is not just the model weights. A system must also store the gradients, the optimizer states (which, for optimizers like Adam, can be twice the size of the weights themselves), and the intermediate activations generated during the forward pass. As noted by Sajal Dash et al., a trillion-parameter model requires tens of terabytes of memory just to hold this state—far exceeding the capacity of any single modern accelerator.
When a model's state exceeds a single device's VRAM, standard data parallelism fails because every device requires a complete copy of the model (Qian Yang et al.). To bypass this constraint, researchers have developed a spectrum of hardware-software co-design strategies that re-engineer how memory is allocated, distributed, and utilized.
Graph Partitioning and Multi-Device Parallelism
The most direct way to handle a model that is too large for one GPU is to slice its computational graph and distribute the pieces across a cluster of devices. This is known as model parallelism.
Historically, partitioning a model required meticulous manual engineering. To automate this, Minjie Wang developed automatic dataflow graph partitioning frameworks like Tofu, which recursively search for optimal ways to divide tensors across devices. Rather than relying on slow, black-box reinforcement learning heuristics to find these splits, Divya Mahajan et al. demonstrated that device placement can be formulated as a structured mathematical optimization problem. By using integer programming and dynamic programming, their approach finds provably optimal graph partitions that minimize communication overhead between devices.
For giant foundation models, systems combine data, tensor, and pipeline parallelism into a unified approach called "3D parallelism." Automated frameworks like Merak (Zhiquan Lai et al.) handle this orchestration out-of-the-box, solving a critical bootstrapping problem: when a giant model cannot fit into a single node's memory, standard compilers cannot even trace its execution graph to figure out how to split it.
Hierarchical Memory Offloading
Instead of scaling out to massive, expensive clusters of GPUs purely to satisfy aggregate memory demands, an alternative approach turns to heterogeneous storage hierarchies. Modern servers possess massive pools of cheaper, slower memory, such as CPU RAM and NVMe SSDs. Hierarchical offloading treats the GPU VRAM as a fast cache and streams data to and from these larger memory pools on demand.
The intuition behind this approach relies on the predictable lifetime of tensors during execution. Not all weights or activations are needed at the same time; a layer's tensors are only active during its specific window in the forward and backward passes. Jiansong Li et al. instrumented GPU memory allocators and discovered that while many small blocks are accessed continuously, large intermediate tensors often sit idle for extended periods. These idle "outlier" blocks can be safely evicted to host memory without stalling execution.
[ GPU VRAM (Fast, Limited) ] <--- High-Speed Streaming ---> [ CPU RAM (Large) ]
^ ^
| |
+----------------- GPUDirect Storage ----------------+
|
v
[ NVMe SSD (Massive) ]
Samyam Rajbhandari pioneered this with ZeRO-Infinity, a system that sharded and offloaded model states across GPU, CPU, and NVMe memory layers, allowing a single GPU node to host models that previously required a cluster. To eliminate the CPU as a communication bottleneck during these transfers, Ziqi Yuan et al. developed TERAIO, which utilizes NVIDIA’s GPUDirect Storage (GDS) to stream inactive tensors directly between the GPU and SSDs. Similarly, Angel-PTM (Xiaonan Nie et al.) abstracts memory into fine-grained "Pages," allowing a unified scheduler to overlap compute with memory prefetching across the hardware hierarchy.
Memory-Aware Compilers and Recomputation
Within the GPU itself, traditional deep learning frameworks often prioritize raw execution speed over memory efficiency, allocating memory greedily and causing abrupt Out-of-Memory (OOM) crashes when limits are breached. Artemev et al. addressed this by developing eXLA, a compiler extension that automatically restructures computational graphs to guarantee execution within user-defined memory bounds.
When memory cannot be freed through better packing alone, systems can trade computation time for memory capacity through rematerialization (or recomputation). Instead of storing every single activation tensor from the forward pass to use during backpropagation, the system discards them and recomputes them on-the-fly when needed. Systems like KARMA (Mohamed Wahib et al.) combine this strategy with CPU data swapping, balancing the cost of redundant GPU calculations against PCIe transfer latencies to maximize throughput.
Compression and Precision Reduction
A complementary strategy to moving or splitting data is simply making the data smaller.
The most widespread realization of this concept is mixed-precision training (Paulius Micikevicius et al.), which proved that deep neural networks can converge successfully when weights and gradients are stored in 16-bit floating-point formats (FP16) rather than traditional 32-bit single precision (FP32). This single shift instantly halved the memory footprint of model states and doubled effective memory bandwidth.
Beyond uniform precision reduction, systems can exploit the mathematical and structural properties of neural network data:
- Activation Sparsity: Because functions like ReLU output many zeros, activation matrices are highly sparse. Minsoo Rhu et al. developed cDMA, a hardware-based compressing DMA engine that compresses activations on-the-fly before offloading them over the PCIe bus, significantly reducing communication bottlenecks.
- Exponent Entropy: Floating-point numbers often contain redundant information. Yongchang Hao et al. introduced NeuZip, a method that dynamically compresses the low-entropy exponent bits of neural network weights during training and inference, slashing memory consumption by over 50% with zero loss in model accuracy.
By combining these multi-layered approaches—automated graph partitioning, hierarchical offloading, memory-aware compilation, and dynamic compression—the deep learning systems ecosystem continues to push past the physical constraints of silicon, ensuring that model scale is dictated by algorithmic imagination rather than hardware memory limits.
Key Foundational Work
Large Scale Distributed Deep Networks (2012) — contributed 2 concept elements to this direction.
Database Meets Deep Learning: Challenges and Opportunities (2016) — contributed 1 concept elements to this direction.