Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters

WenZheng Zhang, Yang Hu, Jing Shi, Xiaoying BaiAAAI 2025 · DOI

TL;DR

Training massive deep neural networks (DNNs) on clusters of GPUs is essential but challenging due to the increasing diversity of GPU hardware. This paper introduces "Poplar," a system that efficiently utilizes heterogeneous GPU clusters by extending the ZeRO optimizer with novel heterogeneity-aware profiling and batch allocation strategies, achieving significant speedups over existing methods.

What Problem Does This Paper Address?

The rapid release cycle of GPUs leads to clusters often comprising a mix of devices with varying compute capabilities, memory capacities, and quantities. Training large DNNs on these heterogeneous clusters is inefficient with current methods, which typically assume homogeneous hardware. This leads to significant GPU idle time during synchronization, as faster GPUs wait for slower ones. Existing approaches to address this either require complex manual configuration or struggle to adapt to the diverse heterogeneity found in real-world clusters, especially when dealing with non-uniform numbers of GPUs. The paper aims to bridge this gap by enabling efficient and automated distributed DNN training on such mixed-hardware environments.

Methods

Poplar extends the Zero Redundancy Optimizer (ZeRO) framework to handle heterogeneous GPUs. Its core innovation lies in two phases:

  • Online Profiling: Poplar first profiles each GPU to understand its unique characteristics. This involves:
  • Memory Capacity Measurement: Automatically determining the maximum batch size (mbs) a GPU can handle without out-of-memory (OOM) errors. This is crucial because manual configuration is tedious and complex in heterogeneous settings.
  • Compute Capability Measurement: Instead of relying solely on theoretical FLOPs, Poplar measures the actual time taken for a GPU to perform a single iteration (forward and backward passes) with varying batch sizes. This provides a more accurate representation of performance.
  • Offline Analyzing: Based on the profiling data, Poplar employs a search algorithm to find an optimal configuration. This involves:
  • Batch Allocation: A novel method to assign micro-batch sizes to each GPU to ensure load balancing and minimize idle time. This considers both compute and memory heterogeneity.
  • Automated Parallelism: Poplar automates the process of determining the optimal parallelism strategy and batch sizes, eliminating the need for expert intervention or manual tuning.

The system is designed to be decoupled from the model training code, allowing easy integration with existing DNN training pipelines.

Datasets

The paper uses the wikitext-2-v1 dataset for evaluating the performance of their system.

Key Insights and Ideas

  • Beyond FLOPs: Relying solely on theoretical FLOPs for GPU compute capability is insufficient. Actual runtime measurements, considering memory capacity and other computational overheads, provide a more accurate picture for heterogeneous systems.
  • Automated Heterogeneity Awareness: Manual configuration of heterogeneous GPU clusters is impractical. Poplar's automated profiling and search algorithms are key to unlocking the potential of these diverse hardware setups.
  • Fine-grained Batch Allocation: The core of Poplar's efficiency comes from its ability to dynamically allocate micro-batch sizes to individual GPUs based on their specific capabilities, rather than enforcing uniform batch sizes across the cluster.
  • ZeRO Extension: Extending the popular ZeRO framework, which is known for its memory efficiency without code refactoring, to handle heterogeneity makes Poplar a practical solution for many researchers and practitioners.

Key Results

Poplar demonstrates significant performance improvements across various heterogeneous GPU clusters and ZeRO stages. Specifically, the paper reports:

  • Training Throughput Improvement: Poplar achieves a training throughput improvement of 1.02x to 3.92x over current state-of-the-art heterogeneous training systems.
  • Performance Across Scenarios: The system performs well in scenarios with differing memory capacities, differing compute capabilities, and combinations thereof.
  • ZeRO Stage Benefits: Improvements are observed across all ZeRO stages, with particularly substantial gains in ZeRO-2 and ZeRO-3 due to Poplar's ability to reduce gradient accumulation steps and communication volume.
  • Model Size Impact: Poplar shows strong performance on both smaller (0.5B parameters) and larger (1.1B parameters) models, with even greater relative gains on larger models and BERT compared to Llama.

Limitations and Future Work

  • Network Bottleneck: The authors acknowledge that their current method does not explicitly address heterogeneous networking, which can present a bottleneck in distributed training. They plan to include network considerations in future work.
  • PCIe Focus: All experiments were conducted on PCIe interconnects, and the paper mentions this as a practical consideration.
  • Future Work: The paper suggests that future work will focus on incorporating network link heterogeneity into their system.

Summary updated

BibTeX
@inproceedings{zhang2025poplar,
  author = {WenZheng Zhang and Yang Hu and Jing Shi and Xiaoying Bai},
  title = {{Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i21.34417},
}
Figure