TL;DR
ZeRO-Offload brings billion-scale deep learning model training to commodity hardware by intelligently partitioning data and computation between GPUs and host CPUs. It enables training models with up to 13 billion parameters on a single 32 GB GPU—a 10× increase over native PyTorch—without model refactoring or sacrificing compute efficiency.
What Problem Does This Paper Address?
State-of-the-art Transformer models (e.g., GPT-3, Megatron-LM) require massive amounts of memory. Under standard 16-bit mixed-precision training with the Adam optimizer, model states (fp16 weights, fp16 gradients, fp32 master weights, momentum, and variance) consume at least $16 \times M$ bytes for an $M$-parameter model. A 10-billion parameter model requires over 160 GB just for model states—far exceeding the memory of high-end GPUs.
Existing solutions like model parallelism, pipeline parallelism, or ZeRO partition model states across multi-GPU clusters, but they still require expensive clusters (e.g., $100K+ DGX-2 nodes) to provide enough aggregated GPU memory. Prior heterogeneous training frameworks that offload data to host CPU memory predominantly focus on convolutional networks (where activations, not model states, dominate memory) and merely use the CPU as passive storage, causing high PCIe transfer overhead and CPU-GPU synchronization bottlenecks.
Methods
ZeRO-Offload designs a unique, communication-optimal schedule that splits both memory and compute across the GPU and CPU:
GPU CPU
┌───────────────────────────────┐ ┌───────────────────────────────┐
│ • fp16 Parameters │ │ • fp16 Gradients (offloaded) │
│ • Forward & Backward Passes │ PCIe │ • fp32 Parameters & Optim. │
│ (O(MB) Compute) │◄────►│ States (Momentum, Variance) │
│ │ │ • Adam Optimizer Step │
└───────────────────────────────┘ │ (O(M) Compute) │
└───────────────────────────────┘
- First-Principles Dataflow Partitioning: Heavy $O(MB)$ compute (forward and backward passes, where $B$ is batch size) remains on the GPU. The lighter $O(M)$ compute (Adam parameter updates) and bulky fp32 optimizer states are placed on the CPU.
- Minimal PCIe Communication: Only fp16 gradients stream from GPU to CPU, and updated fp16 weights stream back from CPU to GPU. This reaches the theoretical minimum communication volume of $4M$ bytes per training step while yielding an 8× reduction in GPU model-state memory.
- High-Performance CPU-Adam: A custom C++ Adam implementation exploiting AVX-512 SIMD vector instructions, loop unrolling (width 8), OpenMP multithreading, and tiled parameter copy to overlap CPU computation with GPU tensor transfers.
- One-Step Delayed Parameter Update (DPU): For small batch sizes where CPU optimizer execution might bottleneck the GPU, DPU overlaps CPU optimizer updates with the subsequent step's GPU forward/backward pass by tolerating a 1-step gradient staleness (enabled after a brief warmup).
- Symbiosis with ZeRO-2 and Model Parallelism: Combined with ZeRO-2 data parallelism, optimizer state partitions are distributed across host CPUs without replicating communication volume. It can also integrate with Megatron-LM tensor slicing.
- Usability: Packaged directly into Microsoft’s open-source DeepSpeed library, requiring only a few lines of code changes in standard PyTorch pipelines.
Datasets
- Pre-training Workloads: GPT-2–style Transformer architectures configured across different layer/hidden-dimension combinations, spanning 1B to 70B parameters.
- Evaluation / Convergence Benchmarks:
- GPT-2 Pre-training: Evaluated training loss curves over 100K iterations.
- SQuAD v1.1 (Stanford Question Answering Dataset): Used for fine-tuning BERT-Large (336M parameters) to measure downstream convergence and accuracy impact under delayed updates.
Key Insights and Ideas
- Offloading compute reduces communication: Offloading both optimizer computation and optimizer states to the CPU avoids transferring high-precision fp32 states across PCIe. Counterintuitively, using the CPU for compute yields higher overall throughput than using it as passive storage.
- Theoretical optimality: By modeling training as a cyclic dataflow graph, the authors prove that placing fp16 parameters/activations on the GPU and fp32 states on the CPU is the unique optimal configuration achieving minimum PCIe volume ($4M$) and maximum GPU memory savings.
- Data parallelism scales CPU resources: Pairing CPU offloading with ZeRO-2 ensures that as the number of GPUs increases, the aggregate CPU update time decreases linearly because each node's CPU processes a smaller slice of parameters while PCIe communication per link remains constant.
Key Results
- Model Scale:
- Single GPU (32 GB V100): Trains up to 13B parameters (compared to 1.4B on PyTorch DDP and 8B on SwapAdvisor).
- Single DGX-2 (16 GPUs): Trains up to 70B parameters when combined with model parallelism (a 4.5× increase over Megatron-LM alone).
- Throughput:
- Sustains 40 TFLOPS/GPU on a single V100 for a 10B model (native PyTorch peaks at ~30 TFLOPS for a 1.4B model before OOM).
- Outperforms SwapAdvisor by 23% (up to 37%) and L2L by 14% (up to 22%).
- CPU-Adam executes 5× to 6.4× faster than PyTorch’s CPU Adam.
- DPU provides a 1.12×–1.59× speedup for micro-batch sizes of 8 without degrading model convergence or final accuracy (BERT-Large fine-tuning matched the baseline 92.8 F1 score on SQuAD).
- Scalability: Demonstrates near-linear scaling up to 128 GPUs on a 10B parameter GPT-2 model, maintaining over 30 TFLOPS/GPU.
Limitations and Future Work
- Activation Memory Limits: ZeRO-Offload focuses exclusively on model states. Activations are managed via activation checkpointing, but for extremely long context lengths, activation memory can still trigger GPU out-of-memory errors unless combined with activation offloading.
- Crossover Point at Massive GPU Scale: When scaling beyond 64–128 GPUs, aggregate GPU memory becomes large enough that pure GPU-based ZeRO-2 outperforms ZeRO-Offload, as PCIe transfers eventually become a relative liability.
- Warmup Requirements for DPU: One-step delayed updates require running the initial iterations (~40 steps) synchronously to avoid destabilizing early optimization dynamics.
Summary updated