TL;DR
ParDNN is an automated, "non-intrusive" system that partitions massive Deep Neural Networks (DNNs) across multiple GPUs. It allows researchers to train models with billions of parameters—far exceeding the memory of a single device—without manual code rewrites, while achieving faster-than-linear (superlinear) performance gains.
What Problem Does This Paper Address?
We are currently hitting a "memory wall" in Deep Learning. State-of-the-art models are growing in size much faster than the memory capacity of individual GPUs. When a model’s parameters and intermediate activations exceed a GPU's RAM, training simply crashes with an "Out of Memory" (OOM) error.
Existing solutions have significant friction:
- Manual Model Parallelism: Requires engineers to manually split code, which is error-prone and model-specific.
- Reinforcement Learning (RL) Approaches: Can take hours or days just to find a good partitioning strategy.
- Gradient Checkpointing: Saves memory by recomputing data, but slows down training significantly.
The authors introduce ParDNN, a system-level tool that automatically decides which parts of a model's "computational graph" should live on which GPU to satisfy memory constraints while keeping the training speed (the "makespan") as fast as possible.
Methods
ParDNN operates directly on the computational graph (the map of operations like matrix multiplications and convolutions) used by frameworks like TensorFlow. Its strategy involves two main stages:
- Step-1: Partitioning for Speed.
- Graph Slicing: It iteratively identifies the "Critical Path" (the longest chain of dependent operations) and assigns these "primary clusters" to different GPUs.
- LFLAM (Locality-First Lookahead and Level-Aware Mapping): This is a novel algorithm that assigns the remaining smaller operations to GPUs. It prioritizes "locality" (keeping communicating operations on the same GPU) to minimize slow data transfers, but switches to "load balancing" if one GPU is becoming a bottleneck.
- Step-2: Handling Memory Overflow. If the initial speed-focused assignment exceeds a GPU's memory, ParDNN uses a Scheduler Emulator to predict memory usage over time. It then treats the overflow as a "knapsack problem," greedily moving the least critical operations to other GPUs until every device is under its memory limit.
Crucially, ParDNN is non-intrusive: it requires zero changes to the model’s mathematical logic or the underlying operation kernels.
Datasets
The researchers evaluated ParDNN using five diverse, large-scale models across different domains:
- Word-RNN & Char-CRN: Language modeling (using Tiny Shakespeare and Penn Treebank).
- Wide Residual Net (WRN): Image classification (CIFAR-100).
- Transformer: Machine translation (IWSLT’16 German-English).
- Eidetic 3D LSTM (E3D): Video prediction (Moving MNIST).
These models were scaled up to billions of parameters to test the limits of the system.
Key Insights and Ideas
- The "Aha" Moment: System-Level Genericity. Because ParDNN works on the raw computational graph, it doesn't need to "understand" the AI. It treats the model as a math problem of nodes and edges, making it compatible with almost any architecture.
- Superlinear Scaling. One of the most surprising findings is that ParDNN often achieves superlinear scaling. This means that using 4 GPUs might make training more than 4x more efficient than a single GPU. This happens because distributing the model allows for much larger batch sizes, which improves GPU utilization and reduces the relative overhead of storing parameters.
- Efficiency. Unlike RL-based partitioners that take hours, ParDNN can partition a graph with hundreds of thousands of nodes in about 2 minutes on a standard laptop CPU.
Key Results
- Massive Model Support: ParDNN successfully trained a Transformer model with 5.1 billion parameters using only 4 GPUs—a feat impossible with standard data parallelism.
- Performance Gains: It outperformed Mesh-TensorFlow (a popular manual partitioning framework) and provided up to a 2.7x speedup over Gradient Checkpointing.
- Memory Efficiency: For the same number of GPUs, ParDNN enabled the use of 9x larger batch sizes on average compared to standard data-parallel training.
- Low Overhead: The time spent calculating the partition is negligible (seconds to minutes) compared to training times that last days or weeks.
Limitations and Future Work
- Offline Profiling: ParDNN requires an initial "profiling" run to estimate how much time and memory each operation takes. While this is a one-time cost, it adds a step to the workflow.
- Single-Node Focus: The current evaluation focuses on a single machine with multiple GPUs (like an NVIDIA DGX-2). While the authors suggest it can be combined with data parallelism for multi-node clusters, they haven't extensively benchmarked massive multi-node scaling in this specific paper.
- Static vs. Dynamic: ParDNN handles memory statically before training starts. While this avoids runtime overhead, it might not account for highly dynamic graphs where memory usage changes wildly between iterations (though most DNNs are relatively static).
Summary updated