Abstract
As Deep Neural Network (DNN) models grow in complexity and parameter count, they increasingly encounter the "memory wall" of individual GPU devices. We present ParDNN, an automated, non-intrusive system designed to partition massive DNNs across multiple GPUs. By operating directly on the model's computational graph, ParDNN optimizes for both execution speed (makespan) and memory constraints without requiring manual code modifications. Our evaluation demonstrates that ParDNN can successfully train models with up to 5.1 billion parameters on limited hardware, often achieving superlinear performance gains and significantly outperforming existing manual and automated partitioning frameworks.
1. Introduction
The rapid advancement of state-of-the-art Deep Learning models has led to a significant disparity between model size and the memory capacity of individual GPUs. When a model's parameters and intermediate activations exceed available RAM, training typically fails with "Out of Memory" (OOM) errors. Current mitigation strategies, such as manual model parallelism, reinforcement learning (RL) based partitioning, or gradient checkpointing, suffer from high engineering friction, long search times, or significant computational overhead.
ParDNN addresses these challenges by providing a system-level tool that automatically determines an optimal mapping of a model's computational graph onto multiple GPUs. It prioritizes training throughput while strictly adhering to device memory limits.
2. System Architecture and Methodology
ParDNN operates as a "non-intrusive" layer between the high-level model definition (e.g., in TensorFlow) and the distributed execution engine. The workflow consists of transforming the model into a computational graph, profiling its operations, and applying a two-stage partitioning algorithm.
2.1 Two-Stage Partitioning Strategy
The core of ParDNN is a two-step optimization process:
- Partitioning for Speed: The system first identifies the "Critical Path"—the longest chain of dependent operations—and assigns these primary clusters to different GPUs. It then utilizes the Locality-First Lookahead and Level-Aware Mapping (LFLAM) algorithm to assign remaining operations. LFLAM balances the need for data locality (minimizing inter-device communication) with load balancing across available compute resources.
- Memory Constraint Satisfaction: If the initial speed-optimized assignment exceeds any GPU's memory capacity, ParDNN employs a Scheduler Emulator to predict peak memory usage over time. It then treats the overflow as a "knapsack problem," greedily migrating the least critical operations to other GPUs until all memory constraints are met.
3. Experimental Evaluation
ParDNN was evaluated using five diverse, large-scale models: Word-RNN, Char-CRN, Wide Residual Net (WRN), Transformer, and Eidetic 3D LSTM (E3D). These models were scaled to billions of parameters to test the system's ability to handle extreme memory pressure.
3.1 Key Results
- Scalability: ParDNN successfully enabled the training of a Transformer model with 5.1 billion parameters using only 4 GPUs, a task that is impossible with standard data-parallel approaches.
- Performance: The system achieved up to a 2.7x speedup compared to gradient checkpointing. Notably, ParDNN often exhibited superlinear scaling, where the efficiency gain exceeded the number of GPUs added. This is attributed to the ability to use larger batch sizes, which improves GPU utilization and amortizes parameter storage overhead.
- Efficiency: Unlike RL-based approaches that can take days to find a partition, ParDNN's graph-based algorithm can process hundreds of thousands of nodes in approximately two minutes on a standard CPU.
4. Conclusion
ParDNN provides a robust, automated solution for training massive DNNs on memory-constrained hardware. By treating model partitioning as a graph optimization problem, it offers a generic and efficient alternative to manual parallelism and slow RL-based methods. Future work includes extending the system to support massive multi-node clusters and exploring dynamic graph partitioning for models with variable execution paths.
Summary updated