Introduction
The primary bottleneck in training state-of-the-art deep learning models is often GPU memory rather than raw compute power. As models grow deeper or handle longer sequences, the memory required to store intermediate feature maps for the backward pass scales linearly $O(n)$ with the depth of the network. This paper introduces a systematic approach to reduce this memory complexity to $O(\sqrt{n})$, enabling the training of "ultra-deep" models on standard hardware.
Graph-Based Optimizations
The authors first propose leveraging computation graph analysis to perform static memory optimizations. These include in-place operations, where an output overwrites its input when the input is no longer needed, and memory sharing, which reuses memory blocks for variables with non-overlapping lifetimes. These techniques allow the system to minimize the footprint of a standard gradient graph without altering the underlying computation logic.
Sublinear Memory via Recomputation
The core innovation is a trade-off between computation and memory. Instead of storing every intermediate activation, the network is divided into $k$ segments. Only the activations at segment boundaries are stored. During the backward pass, the missing intermediate results are recomputed via a local forward pass within each segment.
By setting $k = \sqrt{n}$, both the boundary storage and the local recomputation costs scale at $O(\sqrt{n})$. This strategy transforms the memory-intensive task of training deep networks into a more computationally intensive one, which is often a favorable trade-off on modern GPUs where data movement and memory capacity are the primary bottlenecks.
Key Results and Impact
The effectiveness of these techniques was demonstrated by training a 1,000-layer ResNet on a single Titan X GPU. While a standard implementation would require 48GB of VRAM—far exceeding the hardware's 12GB capacity—the optimized approach used only 7GB. This reduction of nearly 85% came with only a 30% increase in training time.
Similar benefits were observed in LSTMs, where memory usage was reduced fourfold for long sequences. This work paved the way for "gradient checkpointing," a feature now standard in major deep learning frameworks like PyTorch and TensorFlow, allowing researchers to push the boundaries of model depth without requiring specialized, high-memory hardware.
Summary updated