Scaling GRPC Tensorflow on 512 nodes of Cori Supercomputer

Amrita Mathuriya, Thorsten Kurth, Vivek Rane, Mustafa Mustafa, Lei Shao, Debbie Bard, Prabhat, Victor W. LeearXiv 2017 · arXiv

Introduction

Deep learning models are increasingly complex, necessitating distributed training across massive compute clusters. However, frameworks like TensorFlow, originally designed for data center environments, often encounter significant performance hurdles when ported to high-performance computing (HPC) systems. This study, a collaboration between Intel and Lawrence Berkeley National Laboratory, investigates the scaling limits of standard distributed TensorFlow using the Google Remote Procedure Call (GRPC) protocol on the Cori supercomputer.

Methodology

The researchers evaluated two distinct neural network architectures to characterize scaling behavior:

  1. ResNet-50: A computationally intensive model with approximately 25.5 million parameters, benchmarked using the ImageNet (ILSVRC 2012) dataset.
  2. HEP-CNN: A relatively "shallow" scientific network with fewer than 1 million parameters, used for classifying high-energy physics data.

Experiments were conducted on up to 512 Intel Xeon Phi (Knights Landing) nodes of the Cori supercomputer, utilizing the Cray Aries high-speed interconnect. The team employed Synchronous Stochastic Gradient Descent (SGD) and Intel’s Math Kernel Library (MKL) for CPU optimization. To isolate communication overhead, "dummy data" was used to bypass disk I/O bottlenecks.

Architectural Bottlenecks

The study identifies two primary constraints in the standard TensorFlow distributed architecture:

The Parameter Server (PS) Bottleneck

In a Parameter Server architecture, "Worker" nodes compute gradients and transmit them to centralized "Parameter Servers" for averaging and weight updates. As the worker count increases, the traffic directed at each PS grows linearly, creating a communication "hotspot." The authors found that simply increasing the number of Parameter Servers does not resolve the issue due to structural load imbalances; for instance, in ResNet-50, 99% of parameters reside in just 54 tensors, meaning many PS nodes remain underutilized while others are saturated.

GRPC vs. HPC Interconnects

While GRPC is a robust general-purpose communication tool, it is not optimized for specialized HPC hardware like the Aries interconnect. The researchers observed a 5-6x performance gap between GRPC-mediated communication and the theoretical peak bandwidth of the Cori system.

Experimental Results

The scaling performance varied significantly based on model size and complexity.

Scaling efficiency for ResNet-50 and HEP-CNN models across different configurations.

As shown in the figure above, ResNet-50 maintained acceptable scaling (>80% efficiency) up to 128 workers with 32 Parameter Servers. However, at 512 workers, efficiency plummeted to 23%. In contrast, the lightweight HEP-CNN model demonstrated excellent weak scaling, maintaining over 80% efficiency up to 256 nodes even with a single parameter server. This suggests that the "scaling wall" is primarily a challenge for large-scale models with high parameter counts.

Conclusion and Future Directions

The centralized Parameter Server model and the overhead of the GRPC protocol represent fundamental barriers to scaling "Big AI" on supercomputers. To achieve efficient training on thousands of nodes, the authors advocate for a transition toward decentralized communication strategies. Future work should focus on integrating Message Passing Interface (MPI) and collective communication algorithms, such as Ring All-Reduce or Tree-Reduction, which distribute the communication load more equitably across the entire cluster.

Summary updated

Abstract

We explore scaling of the standard distributed Tensorflow with GRPC primitives on up to 512 Intel Xeon Phi (KNL) nodes of Cori supercomputer with synchronous stochastic gradient descent (SGD), and identify causes of scaling inefficiency at higher node counts. To our knowledge, this is the first exploration of distributed GRPC Tensorflow scalability on a HPC supercomputer at such large scale with synchronous SGD. We studied scaling of two convolution neural networks - ResNet-50, a state-of-the-art deep network for classification with roughly 25.5 million parameters, and HEP-CNN, a shallow topology with less than 1 million parameters for common scientific usages. For ResNet-50, we achieve >80% scaling efficiency on up to 128 workers, using 32 parameter servers (PS tasks) with a steep decline down to 23% for 512 workers using 64 PS tasks. Our analysis of the efficiency drop points to low network bandwidth utilization due to combined effect of three factors. (a) Heterogeneous distributed parallelization algorithm which uses PS tasks as centralized servers for gradient averaging is suboptimal for utilizing interconnect bandwidth. (b) Load imbalance among PS tasks hinders their efficient scaling. (c) Underlying communication primitive GRPC is currently inefficient on Cori high-speed interconnect. The HEP-CNN demands less interconnect bandwidth, and shows >80% weak scaling efficiency for up to 256 nodes with only 1 PS task. Our findings are applicable to other deep learning networks. Big networks with millions of parameters stumble upon the issues discussed here. Shallower networks like HEP-CNN with relatively lower number of parameters can efficiently enjoy weak scaling even with a single parameter server.
BibTeX
@misc{mathuriya2017scaling,
  author = {Amrita Mathuriya and Thorsten Kurth and Vivek Rane and Mustafa Mustafa and Lei Shao and Debbie Bard and Prabhat and Victor W. Lee},
  title = {{Scaling GRPC Tensorflow on 512 nodes of Cori Supercomputer}},
  year = {2017},
  eprint = {1712.09388},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1712.09388},
}
Figure