Introduction
Deep learning models are increasingly complex, necessitating distributed training across massive compute clusters. However, frameworks like TensorFlow, originally designed for data center environments, often encounter significant performance hurdles when ported to high-performance computing (HPC) systems. This study, a collaboration between Intel and Lawrence Berkeley National Laboratory, investigates the scaling limits of standard distributed TensorFlow using the Google Remote Procedure Call (GRPC) protocol on the Cori supercomputer.
Methodology
The researchers evaluated two distinct neural network architectures to characterize scaling behavior:
- ResNet-50: A computationally intensive model with approximately 25.5 million parameters, benchmarked using the ImageNet (ILSVRC 2012) dataset.
- HEP-CNN: A relatively "shallow" scientific network with fewer than 1 million parameters, used for classifying high-energy physics data.
Experiments were conducted on up to 512 Intel Xeon Phi (Knights Landing) nodes of the Cori supercomputer, utilizing the Cray Aries high-speed interconnect. The team employed Synchronous Stochastic Gradient Descent (SGD) and Intel’s Math Kernel Library (MKL) for CPU optimization. To isolate communication overhead, "dummy data" was used to bypass disk I/O bottlenecks.
Architectural Bottlenecks
The study identifies two primary constraints in the standard TensorFlow distributed architecture:
The Parameter Server (PS) Bottleneck
In a Parameter Server architecture, "Worker" nodes compute gradients and transmit them to centralized "Parameter Servers" for averaging and weight updates. As the worker count increases, the traffic directed at each PS grows linearly, creating a communication "hotspot." The authors found that simply increasing the number of Parameter Servers does not resolve the issue due to structural load imbalances; for instance, in ResNet-50, 99% of parameters reside in just 54 tensors, meaning many PS nodes remain underutilized while others are saturated.
GRPC vs. HPC Interconnects
While GRPC is a robust general-purpose communication tool, it is not optimized for specialized HPC hardware like the Aries interconnect. The researchers observed a 5-6x performance gap between GRPC-mediated communication and the theoretical peak bandwidth of the Cori system.
Experimental Results
The scaling performance varied significantly based on model size and complexity.
As shown in the figure above, ResNet-50 maintained acceptable scaling (>80% efficiency) up to 128 workers with 32 Parameter Servers. However, at 512 workers, efficiency plummeted to 23%. In contrast, the lightweight HEP-CNN model demonstrated excellent weak scaling, maintaining over 80% efficiency up to 256 nodes even with a single parameter server. This suggests that the "scaling wall" is primarily a challenge for large-scale models with high parameter counts.
Conclusion and Future Directions
The centralized Parameter Server model and the overhead of the GRPC protocol represent fundamental barriers to scaling "Big AI" on supercomputers. To achieve efficient training on thousands of nodes, the authors advocate for a transition toward decentralized communication strategies. Future work should focus on integrating Message Passing Interface (MPI) and collective communication algorithms, such as Ring All-Reduce or Tree-Reduction, which distribute the communication load more equitably across the entire cluster.
Summary updated