Workload-aware Automatic Parallelization for Multi-GPU DNN Training

Sungho Shin, Youngmin Jo, Jungwook Choi, Swagath Venkataramani, Vijayalakshmi Srinivasan, Wonyong Sung

Abstract

Deep neural networks (DNNs) have emerged as successful solutions for variety of artificial intelligence applications, but their very large and deep models impose high computational requirements during training. Multi-GPU parallelization is a popular option to accelerate demanding computations in DNN training, but most state-of-the-art multi-GPU deep learning frameworks not only require users to have an in-depth understanding of the implementation of the frameworks themselves, but also apply parallelization in a straight-forward way without optimizing GPU utilization. In this work, we propose a workload-aware auto-parallelization framework (WAP) for DNN training, where the work is automatically distributed to multiple GPUs based on the workload characteristics. We evaluate WAP using TensorFlow with popular DNN benchmarks (AlexNet and VGG-16), and show competitive training throughput compared with the state-of-the-art frameworks, and also demonstrate that WAP automatically optimizes GPU assignment based on the workload's compute requirements, thereby improving energy efficiency.

Introduction

While multi-GPU training is the standard for accelerating deep learning, it remains a "leaky abstraction." Popular frameworks like TensorFlow and PyTorch often require researchers to manually handle the complexities of data distribution and gradient synchronization. This manual effort often leads to boilerplate-heavy code and potential performance degradation if not tuned correctly.

Comparison of user effort across different parallelization frameworks. WAP aims to use the simple Single-GPU code (a) to achieve the performance of complex manual implementations (b, c).

Existing "auto-parallel" tools often blindly distribute workloads to every available GPU. However, if the workload is small (e.g., a small minibatch size), the time spent communicating data between GPUs can exceed the time saved by parallel computation. This results in a "slowdown" despite having more hardware—a phenomenon we call the "More is Less" paradox. WAP (Workload-aware Auto-Parallelization) addresses this by automatically predicting the optimal number of GPUs to use, preventing performance loss and power waste.

The WAP Framework

WAP operates "under the hood" by modifying the TensorFlow core source code (C++), allowing users to write standard single-GPU scripts while the framework handles the distribution. The process follows four main stages:

The WAP system architecture and automated pipeline.
  1. Neural-Net Parser: This component intercepts the high-level Python description of a model and extracts workload details—such as layer types, dimensions, and batch sizes—from the internal dataflow graph.
  2. Workload Analysis Unit (WAU): Acting as the system's "brain," the WAU uses a performance model to estimate total execution time by calculating the sum of computation time and communication overhead for various GPU counts. It then selects the configuration that minimizes total time.
  3. Graph Modifier: Once the optimal GPU count is determined, WAP replicates the necessary nodes in the graph. It also replicates auxiliary nodes (like loss functions) to minimize unnecessary data transfers between devices.
  4. Post-Processing: To maximize communication speed, WAP replaces standard gradient aggregation with NCCL AllReduce, keeping communication overhead linear as more GPUs are added.

Experimental Evaluation

The framework was evaluated using AlexNet and VGG-16 benchmarks on two hardware environments: a 4-GPU workstation and an 8-GPU NVIDIA DGX-1 system.

Key Results

  • Intelligent Allocation: In tests with AlexNet and a small batch size (128), competing frameworks blindly used all 4 available GPUs, resulting in 1,473 images/sec. WAP analyzed the workload, realized a single GPU was more efficient, and achieved 2,560 images/sec.
  • Energy Efficiency: By leaving underutilized GPUs idle in the aforementioned scenario, WAP reduced power consumption by 63% compared to the "blind" multi-GPU approach.
  • Scalability: On the DGX-1 system, WAP showed excellent scaling for VGG-16, nearly matching the ideal linear speedup curve due to its optimized NCCL integration.

Conclusion

WAP demonstrates that automation in deep learning parallelization does not have to come at the cost of peak performance. By making energy and workload characteristics first-class citizens in the optimization process, WAP achieves "zero-effort" scaling that is both faster and more efficient than traditional methods. Future work will focus on extending these automated strategies to model and hybrid parallelization for ultra-large-scale networks.

Summary updated

Abstract

Deep neural networks (DNNs) have emerged as successful solutions for variety of artificial intelligence applications, but their very large and deep models impose high computational requirements during training. Multi-GPU parallelization is a popular option to accelerate demanding computations in DNN training, but most state-of-the-art multi-GPU deep learning frameworks not only require users to have an in-depth understanding of the implementation of the frameworks themselves, but also apply parallelization in a straight-forward way without optimizing GPU utilization. In this work, we propose a workload-aware auto-parallelization framework (WAP) for DNN training, where the work is automatically distributed to multiple GPUs based on the workload characteristics. We evaluate WAP using TensorFlow with popular DNN benchmarks (AlexNet and VGG-16), and show competitive training throughput compared with the state-of-the-art frameworks, and also demonstrate that WAP automatically optimizes GPU assignment based on the workload's compute requirements, thereby improving energy efficiency.
BibTeX
@inproceedings{shin2019workload,
  author = {Sungho Shin and Youngmin Jo and Jungwook Choi and Swagath Venkataramani and Vijayalakshmi Srinivasan and Wonyong Sung},
  title = {{Workload-aware Automatic Parallelization for Multi-GPU DNN Training}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=Wx3Avm8B_Dc},
}
Figure