TL;DR
WAP (Workload-aware Auto-Parallelization) is a framework that automatically scales deep learning training across multiple GPUs without requiring any code changes from the user. By analyzing the neural network's dataflow graph, WAP predicts the optimal number of GPUs to use, preventing the performance degradation and power waste that often occur when small workloads are spread too thin across many processors.
What Problem Does This Paper Address?
While multi-GPU training is the standard for accelerating deep learning, it remains a "leaky abstraction." Popular frameworks like TensorFlow and PyTorch often require researchers to manually handle the complexities of data distribution and gradient synchronization.
Existing "auto-parallel" tools often fall into a trap: they blindly distribute the workload to every available GPU. However, if the workload is small (e.g., a small minibatch size), the time spent communicating data between GPUs can actually exceed the time saved by parallel computation. This results in a "slowdown" despite having more hardware. Currently, the burden is on the engineer to manually profile and decide exactly how many GPUs are "just right" for a specific model and batch size.
Methods
The authors introduce WAP, which operates "under the hood" by modifying the TensorFlow core source code (C++). The process follows four main steps:
- Neural-Net Parser: It intercepts the high-level Python description of a model and extracts the workload details (layer types, dimensions, and batch sizes) from the internal dataflow graph.
- Workload Analysis Unit (WAU): This is the "brain" of the system. It uses a performance model to estimate total execution time ($t_{estimate}$) by calculating the sum of computation time ($t_c$) and synchronization/communication time ($t_s$) for different numbers of GPUs. It then selects the number of GPUs that minimizes this total time.
- Graph Modifier: Once the number of GPUs is chosen, WAP automatically replicates the necessary nodes in the graph. Crucially, it also replicates auxiliary nodes (like loss functions and activation layers) to minimize unnecessary data transfers between devices.
- Post-Processing: To ensure the fastest possible communication, WAP replaces standard gradient aggregation with NCCL AllReduce, which uses a "ring" algorithm to keep communication overhead linear rather than exponential as more GPUs are added.
Datasets
The framework was evaluated using two classic, computationally intensive benchmarks:
- AlexNet: Used to test high-throughput scenarios and sensitivity to minibatch sizes.
- VGG-16: A deeper model used to evaluate how the framework handles more complex dataflow graphs.
Testing was performed on two hardware environments: a "Single Machine" (SM) with 4 NVIDIA TitanXP GPUs and an NVIDIA DGX-1 with 8 Tesla GP100 GPUs connected via NVLink.
Key Insights and Ideas
- The "More is Less" Paradox: The paper quantifies how using more GPUs can be detrimental. For a small AlexNet workload (minibatch 128), using 4 GPUs was significantly slower than using just 1 because the communication overhead "overshadowed" the compute gains.
- Graph-Level Transparency: By implementing WAP at the TensorFlow core level rather than as a Python wrapper, the authors achieve "zero-effort" parallelization. A user can write a standard single-GPU script, and WAP handles the rest.
- Energy as a First-Class Citizen: Most parallelization papers focus solely on speed. WAP highlights that by not using unnecessary GPUs, it can drastically reduce power consumption without sacrificing (and sometimes even improving) throughput.
Key Results
- Competitive Performance: WAP achieved training throughput comparable to TF-Bench, which is a manually hand-optimized benchmark suite. This proves that automation doesn't have to come at the cost of peak performance.
- Intelligent Allocation: In a test with AlexNet and a small batch size, a competing framework (Parallax) blindly used all 4 available GPUs, resulting in 1,473 images/sec. WAP analyzed the workload, realized 1 GPU was better, and achieved 2,560 images/sec.
- Efficiency Gains: In the same scenario mentioned above, WAP reduced power consumption by 63% compared to the "blind" multi-GPU approach by simply leaving underutilized GPUs idle.
- Scalability: On the DGX-1 system, WAP showed excellent scaling for VGG-16, nearly matching the "ideal" linear speedup curve thanks to its optimized NCCL integration.
Limitations and Future Work
The primary limitation acknowledged is that WAP currently focuses on data parallelization (splitting the data across GPUs). While this is the most common strategy, very large models that don't fit into a single GPU's memory require model parallelization (splitting the layers across GPUs). The authors note that implementing automated model and hybrid parallelization strategies is the next step for the framework. Additionally, the performance model currently relies on known execution time estimates; making this model more dynamic or hardware-agnostic could improve its accuracy across different generations of GPUs.
Summary updated