ZeRO-Offload: Democratizing Billion-Scale Model Training

Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He

TL;DR

ZeRO-Offload brings billion-scale deep learning model training to commodity hardware by intelligently partitioning data and computation between GPUs and host CPUs. It enables training models with up to 13 billion parameters on a single 32 GB GPU—a 10× increase over native PyTorch—without model refactoring or sacrificing compute efficiency.

What Problem Does This Paper Address?

State-of-the-art Transformer models (e.g., GPT-3, Megatron-LM) require massive amounts of memory. Under standard 16-bit mixed-precision training with the Adam optimizer, model states (fp16 weights, fp16 gradients, fp32 master weights, momentum, and variance) consume at least $16 \times M$ bytes for an $M$-parameter model. A 10-billion parameter model requires over 160 GB just for model states—far exceeding the memory of high-end GPUs.

Existing solutions like model parallelism, pipeline parallelism, or ZeRO partition model states across multi-GPU clusters, but they still require expensive clusters (e.g., $100K+ DGX-2 nodes) to provide enough aggregated GPU memory. Prior heterogeneous training frameworks that offload data to host CPU memory predominantly focus on convolutional networks (where activations, not model states, dominate memory) and merely use the CPU as passive storage, causing high PCIe transfer overhead and CPU-GPU synchronization bottlenecks.

Methods

ZeRO-Offload designs a unique, communication-optimal schedule that splits both memory and compute across the GPU and CPU:

                GPU                                    CPU
 ┌───────────────────────────────┐      ┌───────────────────────────────┐
 │ • fp16 Parameters             │      │ • fp16 Gradients (offloaded)  │
 │ • Forward & Backward Passes   │ PCIe │ • fp32 Parameters & Optim.    │
 │   (O(MB) Compute)             │◄────►│   States (Momentum, Variance) │
 │                               │      │ • Adam Optimizer Step         │
 └───────────────────────────────┘      │   (O(M) Compute)              │
                                        └───────────────────────────────┘
  • First-Principles Dataflow Partitioning: Heavy $O(MB)$ compute (forward and backward passes, where $B$ is batch size) remains on the GPU. The lighter $O(M)$ compute (Adam parameter updates) and bulky fp32 optimizer states are placed on the CPU.
  • Minimal PCIe Communication: Only fp16 gradients stream from GPU to CPU, and updated fp16 weights stream back from CPU to GPU. This reaches the theoretical minimum communication volume of $4M$ bytes per training step while yielding an 8× reduction in GPU model-state memory.
  • High-Performance CPU-Adam: A custom C++ Adam implementation exploiting AVX-512 SIMD vector instructions, loop unrolling (width 8), OpenMP multithreading, and tiled parameter copy to overlap CPU computation with GPU tensor transfers.
  • One-Step Delayed Parameter Update (DPU): For small batch sizes where CPU optimizer execution might bottleneck the GPU, DPU overlaps CPU optimizer updates with the subsequent step's GPU forward/backward pass by tolerating a 1-step gradient staleness (enabled after a brief warmup).
  • Symbiosis with ZeRO-2 and Model Parallelism: Combined with ZeRO-2 data parallelism, optimizer state partitions are distributed across host CPUs without replicating communication volume. It can also integrate with Megatron-LM tensor slicing.
  • Usability: Packaged directly into Microsoft’s open-source DeepSpeed library, requiring only a few lines of code changes in standard PyTorch pipelines.

Datasets

  • Pre-training Workloads: GPT-2–style Transformer architectures configured across different layer/hidden-dimension combinations, spanning 1B to 70B parameters.
  • Evaluation / Convergence Benchmarks:
  • GPT-2 Pre-training: Evaluated training loss curves over 100K iterations.
  • SQuAD v1.1 (Stanford Question Answering Dataset): Used for fine-tuning BERT-Large (336M parameters) to measure downstream convergence and accuracy impact under delayed updates.

Key Insights and Ideas

  • Offloading compute reduces communication: Offloading both optimizer computation and optimizer states to the CPU avoids transferring high-precision fp32 states across PCIe. Counterintuitively, using the CPU for compute yields higher overall throughput than using it as passive storage.
  • Theoretical optimality: By modeling training as a cyclic dataflow graph, the authors prove that placing fp16 parameters/activations on the GPU and fp32 states on the CPU is the unique optimal configuration achieving minimum PCIe volume ($4M$) and maximum GPU memory savings.
  • Data parallelism scales CPU resources: Pairing CPU offloading with ZeRO-2 ensures that as the number of GPUs increases, the aggregate CPU update time decreases linearly because each node's CPU processes a smaller slice of parameters while PCIe communication per link remains constant.

Key Results

  • Model Scale:
  • Single GPU (32 GB V100): Trains up to 13B parameters (compared to 1.4B on PyTorch DDP and 8B on SwapAdvisor).
  • Single DGX-2 (16 GPUs): Trains up to 70B parameters when combined with model parallelism (a 4.5× increase over Megatron-LM alone).
  • Throughput:
  • Sustains 40 TFLOPS/GPU on a single V100 for a 10B model (native PyTorch peaks at ~30 TFLOPS for a 1.4B model before OOM).
  • Outperforms SwapAdvisor by 23% (up to 37%) and L2L by 14% (up to 22%).
  • CPU-Adam executes 5× to 6.4× faster than PyTorch’s CPU Adam.
  • DPU provides a 1.12×–1.59× speedup for micro-batch sizes of 8 without degrading model convergence or final accuracy (BERT-Large fine-tuning matched the baseline 92.8 F1 score on SQuAD).
  • Scalability: Demonstrates near-linear scaling up to 128 GPUs on a 10B parameter GPT-2 model, maintaining over 30 TFLOPS/GPU.

Limitations and Future Work

  • Activation Memory Limits: ZeRO-Offload focuses exclusively on model states. Activations are managed via activation checkpointing, but for extremely long context lengths, activation memory can still trigger GPU out-of-memory errors unless combined with activation offloading.
  • Crossover Point at Massive GPU Scale: When scaling beyond 64–128 GPUs, aggregate GPU memory becomes large enough that pure GPU-based ZeRO-2 outperforms ZeRO-Offload, as PCIe transfers eventually become a relative liability.
  • Warmup Requirements for DPU: One-step delayed updates require running the initial iterations (~40 steps) synchronously to avoid destabilizing early optimization dynamics.

Summary updated

Abstract

Large-scale model training has been a playing ground for a limited few requiring complex model refactoring and access to prohibitively expensive GPU clusters. ZeRO-Offload changes the large model training landscape by making large model training accessible to nearly everyone. It can train models with over 13 billion parameters on a single GPU, a 10x increase in size compared to popular framework such as PyTorch, and it does so without requiring any model change from the data scientists or sacrificing computational efficiency. ZeRO-Offload enables large model training by offloading data and compute to CPU. To preserve compute efficiency, it is designed to minimize the data movement to/from GPU, and reduce CPU compute time while maximizing memory savings on GPU. As a result, ZeRO-Offload can achieve 40 TFlops/GPU on a single NVIDIA V100 GPU for 10B parameter model compared to 30TF using PyTorch alone for a 1.4B parameter model, the largest that can be trained without running out of memory. ZeRO-Offload is also designed to scale on multiple-GPUs when available, offering near linear speedup on up to 128 GPUs. Additionally, it can work together with model parallelism to train models with over 70 billion parameters on a single DGX-2 box, a 4.5x increase in model size compared to using model parallelism alone. By combining compute and memory efficiency with ease-of-use, ZeRO-Offload democratizes large-scale model training making it accessible to even data scientists with access to just a single GPU.
BibTeX
@inproceedings{ren2021zero,
  author = {Jie Ren and Samyam Rajbhandari and Reza Yazdani Aminabadi and Olatunji Ruwase and Shuangyan Yang and Minjia Zhang and Dong Li and Yuxiong He},
  title = {{ZeRO-Offload: Democratizing Billion-Scale Model Training}},
  booktitle = {USENIX ATC},
  year = {2021},
  url = {https://openreview.net/forum?id=qXFQtGMHRa},
}

Loading related content...

Figure