ZeRO-Offload: Democratizing Billion-Scale Model Training

Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He

TL;DR

ZeRO-Offload is a novel technique that democratizes large-scale deep learning model training by offloading model states (parameters, gradients, and optimizer states) from expensive GPUs to readily available CPUs. This allows training models up to 10x larger on a single GPU without sacrificing computational efficiency, and scales effectively to multiple GPUs.

What Problem Does This Paper Address?

Training massive deep learning models, especially large language models (LLMs), has become prohibitively expensive and complex. These models have grown exponentially in size, with the largest now boasting hundreds of billions of parameters. This growth in size directly translates to immense memory requirements that far exceed the capacity of even high-end GPUs. Consequently, only a select few organizations with access to massive GPU clusters can afford to train these state-of-the-art models. Existing distributed training techniques like model and data parallelism, while powerful, still demand significant GPU resources, making them inaccessible to many researchers and data scientists. The core problem is the accessibility barrier to training large-scale models due to hardware limitations and the complexity of distributed training.

Methods

ZeRO-Offload tackles this problem by introducing a heterogeneous training approach that leverages both GPUs and CPUs. The key innovation lies in its "optimal offload strategy," which intelligently partitions the deep learning training process into GPU-bound and CPU-bound tasks.

  • Offloading Model States: Instead of keeping all model states (parameters, gradients, and optimizer states) on the GPU, ZeRO-Offload moves the gradients and optimizer states to the CPU. This significantly frees up GPU memory.
  • CPU Compute Utilization: Crucially, ZeRO-Offload doesn't just offload data; it also utilizes CPU compute for certain operations, specifically the optimizer updates. This is done in a way that minimizes communication overhead and prevents the CPU from becoming a bottleneck.
  • Optimized CPU Optimizer: To further enhance CPU performance, the authors developed a highly optimized CPU Adam optimizer that exploits SIMD vector instructions, loop unrolling, and OMP multithreading, achieving significant speedups over standard PyTorch implementations.
  • Delayed Parameter Update (DPU): To hide the CPU computation and communication latency, ZeRO-Offload employs a one-step delayed parameter update strategy. This overlaps CPU optimizer steps with GPU forward/backward passes, effectively masking the overhead without impacting model accuracy.
  • Integration with ZeRO Data Parallelism: ZeRO-Offload seamlessly integrates with ZeRO-powered data parallelism. This combination allows for efficient scaling across multiple GPUs by maintaining a single copy of optimizer states on the CPU, regardless of the data parallelism degree.
  • Model Parallelism Compatibility: It can also be combined with model parallelism to train even larger models.

Datasets

The paper does not explicitly mention specific datasets used for training the models. However, it references common benchmarks and model architectures used in the field, such as GPT-2 and BERT, for evaluation purposes.

Key Insights and Ideas

  • CPU is More Than Just Storage: The paper demonstrates that CPUs can be effectively utilized for computation in deep learning training, not just as a passive memory extension. By intelligently offloading and utilizing CPU compute, significant efficiency gains can be achieved.
  • Optimal Offloading is Key: Not all offloading strategies are equal. ZeRO-Offload's first-principles analysis to find the "unique optimal" strategy for partitioning computation and data between CPU and GPU is a critical insight. This strategy minimizes communication and CPU computation while maximizing memory savings.
  • Democratization is Achievable: The core idea is that by making large model training accessible on more common hardware (single GPUs with CPU offload), the field can be democratized, allowing more researchers and developers to contribute.
  • DPU is a Clever Trick: The delayed parameter update is a smart way to hide latency without compromising accuracy, a common challenge in heterogeneous computing.

Key Results

  • Model Scale: ZeRO-Offload enables training models up to 13 billion parameters on a single GPU, a 10x increase compared to standard PyTorch. It also allows training models up to 70 billion parameters on a single DGX-2 node when combined with model parallelism, a 4.5x increase over model parallelism alone.
  • Efficiency: Achieved 40 TFLOPS/GPU on a single NVIDIA V100 for a 10B parameter model, compared to 30 TFLOPS for a 1.2B parameter model with PyTorch alone.
  • Scalability: Demonstrates near-linear speedup on up to 128 GPUs.
  • CPU-Adam Speedup: The optimized CPU Adam optimizer offers over 6x speedup compared to PyTorch's CPU implementation.
  • DPU Throughput Improvement: The Delayed Parameter Update (DPU) technique improves training throughput by 1.12–1.59x without impacting model convergence or accuracy.

Limitations and Future Work

  • CPU Bottleneck with Small Batch Sizes: While the optimized CPU optimizer and DPU help, the paper acknowledges that CPU computation can still become a bottleneck with very small batch sizes.
  • Focus on Transformer Models: The evaluation primarily focuses on Transformer-based models. While the principles are likely generalizable, further evaluation on other model architectures might be beneficial.
  • Activation Memory: ZeRO-Offload primarily addresses model state memory bottlenecks. Activation memory, another significant bottleneck for CNNs, is not directly addressed by this offload strategy. The authors suggest that existing techniques for activation memory can be used in conjunction with ZeRO-Offload.
  • Future Work: The paper implicitly suggests future work in further optimizing CPU-GPU communication, exploring more sophisticated offloading strategies, and extending support to a wider range of hardware and model architectures. The open-source release of ZeRO-Offload is a significant step towards enabling further research and development in this area.

Summary updated

Abstract

Large-scale model training has been a playing ground for a limited few requiring complex model refactoring and access to prohibitively expensive GPU clusters. ZeRO-Offload changes the large model training landscape by making large model training accessible to nearly everyone. It can train models with over 13 billion parameters on a single GPU, a 10x increase in size compared to popular framework such as PyTorch, and it does so without requiring any model change from the data scientists or sacrificing computational efficiency. ZeRO-Offload enables large model training by offloading data and compute to CPU. To preserve compute efficiency, it is designed to minimize the data movement to/from GPU, and reduce CPU compute time while maximizing memory savings on GPU. As a result, ZeRO-Offload can achieve 40 TFlops/GPU on a single NVIDIA V100 GPU for 10B parameter model compared to 30TF using PyTorch alone for a 1.4B parameter model, the largest that can be trained without running out of memory. ZeRO-Offload is also designed to scale on multiple-GPUs when available, offering near linear speedup on up to 128 GPUs. Additionally, it can work together with model parallelism to train models with over 70 billion parameters on a single DGX-2 box, a 4.5x increase in model size compared to using model parallelism alone. By combining compute and memory efficiency with ease-of-use, ZeRO-Offload democratizes large-scale model training making it accessible to even data scientists with access to just a single GPU.
BibTeX
@inproceedings{ren2021zero,
  author = {Jie Ren and Samyam Rajbhandari and Reza Yazdani Aminabadi and Olatunji Ruwase and Shuangyan Yang and Minjia Zhang and Dong Li and Yuxiong He},
  title = {{ZeRO-Offload: Democratizing Billion-Scale Model Training}},
  booktitle = {USENIX ATC},
  year = {2021},
  url = {https://openreview.net/forum?id=qXFQtGMHRa},
}
Figure