Training Deep Nets with Sublinear Memory Cost

Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin

Introduction

The primary bottleneck in training state-of-the-art deep learning models is often GPU memory rather than raw compute power. As models grow deeper or handle longer sequences, the memory required to store intermediate feature maps for the backward pass scales linearly $O(n)$ with the depth of the network. This paper introduces a systematic approach to reduce this memory complexity to $O(\sqrt{n})$, enabling the training of "ultra-deep" models on standard hardware.

Graph-Based Optimizations

The authors first propose leveraging computation graph analysis to perform static memory optimizations. These include in-place operations, where an output overwrites its input when the input is no longer needed, and memory sharing, which reuses memory blocks for variables with non-overlapping lifetimes. These techniques allow the system to minimize the footprint of a standard gradient graph without altering the underlying computation logic.

Figure 1: Illustration of graph-based optimizations including in-place operations and memory sharing.

Sublinear Memory via Recomputation

The core innovation is a trade-off between computation and memory. Instead of storing every intermediate activation, the network is divided into $k$ segments. Only the activations at segment boundaries are stored. During the backward pass, the missing intermediate results are recomputed via a local forward pass within each segment.

By setting $k = \sqrt{n}$, both the boundary storage and the local recomputation costs scale at $O(\sqrt{n})$. This strategy transforms the memory-intensive task of training deep networks into a more computationally intensive one, which is often a favorable trade-off on modern GPUs where data movement and memory capacity are the primary bottlenecks.

Figure 6: Comparison between a normal gradient graph and a memory-optimized gradient graph showing shared memory allocations and control dependencies for recomputation.

Key Results and Impact

The effectiveness of these techniques was demonstrated by training a 1,000-layer ResNet on a single Titan X GPU. While a standard implementation would require 48GB of VRAM—far exceeding the hardware's 12GB capacity—the optimized approach used only 7GB. This reduction of nearly 85% came with only a 30% increase in training time.

Similar benefits were observed in LSTMs, where memory usage was reduced fourfold for long sequences. This work paved the way for "gradient checkpointing," a feature now standard in major deep learning frameworks like PyTorch and TensorFlow, allowing researchers to push the boundaries of model depth without requiring specialized, high-memory hardware.

Summary updated

Abstract

We propose a systematic approach to reduce the memory consumption of deep neural network training. Specifically, we design an algorithm that costs O(sqrt(n)) memory to train a n layer network, with only the computational cost of an extra forward pass per mini-batch. As many of the state-of-the-art models hit the upper bound of the GPU memory, our algorithm allows deeper and more complex models to be explored, and helps advance the innovations in deep learning research. We focus on reducing the memory cost to store the intermediate feature maps and gradients during training. Computation graph analysis is used for automatic in-place operation and memory sharing optimizations. We show that it is possible to trade computation for memory - giving a more memory efficient training algorithm with a little extra computation cost. In the extreme case, our analysis also shows that the memory consumption can be reduced to O(log n) with as little as O(n log n) extra cost for forward computation. Our experiments show that we can reduce the memory cost of a 1,000-layer deep residual network from 48G to 7G with only 30 percent additional running time cost on ImageNet problems. Similarly, significant memory cost reduction is observed in training complex recurrent neural networks on very long sequences.
BibTeX
@misc{chen2016training,
  author = {Tianqi Chen and Bing Xu and Chiyuan Zhang and Carlos Guestrin},
  title = {{Training Deep Nets with Sublinear Memory Cost}},
  year = {2016},
  eprint = {1604.06174},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1604.06174},
}

Loading related content...

Figure