Is Network the Bottleneck of Distributed Training?

Zhen Zhang, Chaokun Chang, Haibin Lin, Yida Wang, Raman Arora, Xin Jin

TL;DR

This paper investigates whether network bandwidth is the true bottleneck in distributed deep learning training. Through empirical measurements and simulations, they find that current systems achieve poor scaling not due to slow networks, but due to underutilized network capacity caused by inefficient transport implementations. They argue that optimizing network transport is key to achieving near-linear scaling, potentially eliminating the need for aggressive gradient compression.

What Problem Does This Paper Address?

Training increasingly large and complex deep neural networks (DNNs) requires massive computational resources. To meet this demand, distributed training across multiple machines and GPUs has become essential. The ultimate goal is "linear scalability," meaning that using n machines should yield n times the throughput of a single machine. However, achieving this ideal is challenging, and a common belief is that the network communication required to synchronize gradients between workers becomes the bottleneck as the number of workers increases. This paper aims to systematically understand if the network is indeed the bottleneck, to what extent, and to identify the root cause of this limitation.

Methods

The authors employ a two-pronged approach:

  1. Empirical Measurement Study: They conduct experiments on Amazon Web Services (AWS) using high-end instances (p3dn.24xlarge with 8 NVIDIA V100 GPUs and 100 Gbps networking). They use the Horovod framework with PyTorch to train representative DNN models (ResNet50, ResNet101, VGG16) with varying numbers of workers (from 16 to 64 GPUs). They measure throughput and analyze CPU and network utilization under different network bandwidths.
  2. "What-If" Analysis (Simulation): To explore scenarios where the network is fully utilized, they develop a simulation framework. This involves detailed logging of gradient computation times and then simulating the communication phase under the assumption of 100% network utilization. This allows them to evaluate the potential scaling factor if the network transport were perfectly efficient. They also use this simulation to assess the impact of application-layer optimizations like gradient compression.

Datasets

The experiments were conducted using the ImageNet dataset.

Key Insights and Ideas

  • Network Underutilization is the Real Culprit: Contrary to popular belief, the network itself isn't saturated. Even with 100 Gbps links, the actual data transfer for gradients is significantly less, indicating that the network transport layer is not efficiently utilizing the available bandwidth.
  • CPU is Not the Bottleneck: The low network utilization is not due to the CPU struggling to keep up, as CPU utilization remains low even at high network speeds.
  • Full Network Utilization Yields Near-Linear Scaling: Their "what-if" analysis suggests that if the network could be fully utilized, distributed training could achieve scaling factors close to 100% for all tested models.
  • Gradient Compression's Diminishing Returns: For high-speed networks (like 100 Gbps), aggressive gradient compression is largely unnecessary if the network is fully utilized. For slower networks (like 10 Gbps), moderate compression (2x-5x) is sufficient for near-linear scaling.
  • Network-Layer Optimizations are Preferred: Optimizations at the network transport level are more desirable than application-layer techniques like gradient compression because they are transparent to the application, require no code changes, and do not risk impacting model accuracy or convergence.

Key Results

  • Measured Scaling Factor: With 64 workers on AWS, the system achieved a scaling factor of only around 60% for VGG16, and no more than 76% for ResNet50.
  • Network Utilization: At 100 Gbps, network utilization for gradient transfer was significantly below capacity, often using less than 32 Gbps.
  • CPU Utilization: CPU utilization was low (14%-25%) during the communication phase, ruling it out as a bottleneck.
  • Simulated Scaling Factor (Full Utilization): Under simulated full network utilization, scaling factors approached 100% for all models, even with 64 GPUs.
  • Gradient Compression Impact: On a 10 Gbps network, a 2x-5x compression ratio was sufficient to achieve near-linear scaling when the network was fully utilized. On a 100 Gbps network, compression provided minimal benefit under full utilization.

Limitations and Future Work

Limitations:

  • The study was conducted on a specific cloud provider (AWS) and with specific hardware (NVIDIA V100 GPUs).
  • The analysis focused on the all-reduce communication strategy and a limited set of popular DNN models.

Future Work:

  • Expand the measurement and analysis to a wider range of models (e.g., RNNs, BERT), devices (other GPUs, TPUs), and training strategies (e.g., parameter server, asynchronous training).
  • Investigate other application-layer and system-layer optimizations (e.g., ByteScheduler, SwitchML) to evaluate their potential benefits when the network is fully utilized.
  • The authors strongly advocate for the networking community to develop high-performance network transport solutions that can fully utilize modern high-speed networks to achieve linear scale-out. This includes exploring kernel-bypass technologies like AWS Elastic Fabric Adapter (EFA).

Summary updated

Abstract

Recently there has been a surge of research on improving the communication efficiency of distributed training. However, little work has been done to systematically understand whether the network is the bottleneck and to what extent. In this paper, we take a first-principles approach to measure and analyze the network performance of distributed training. As expected, our measurement confirms that communication is the component that blocks distributed training from linear scale-out. However, contrary to the common belief, we find that the network is running at low utilization and that if the network can be fully utilized, distributed training can achieve a scaling factor of close to one. Moreover, while many recent proposals on gradient compression advocate over 100x compression ratio, we show that under full network utilization, there is no need for gradient compression in 100 Gbps network. On the other hand, a lower speed network like 10 Gbps requires only 2x--5x gradients compression ratio to achieve almost linear scale-out. Compared to application-level techniques like gradient compression, network-level optimizations do not require changes to applications and do not hurt the performance of trained models. As such, we advocate that the real challenge of distributed training is for the network community to develop high-performance network transport to fully utilize the network capacity and achieve linear scale-out.
BibTeX
@inproceedings{zhang2020is,
  author = {Zhen Zhang and Chaokun Chang and Haibin Lin and Yida Wang and Raman Arora and Xin Jin},
  title = {{Is Network the Bottleneck of Distributed Training?}},
  booktitle = {Netai@sigcomm},
  year = {2020},
  doi = {10.1145/3405671.3405810},
}
Figure