Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey

Feng Liang, Zhen Zhang, Haifeng Lu, Victor C. M. Leung, Yanyi Guo, Xiping HuarXiv 2024 · arXiv

TL;DR

As deep learning models scale to billions of parameters, the time spent moving data between GPUs has become a massive performance bottleneck. This comprehensive survey maps out the state-of-the-art (2018–2023) in communication-efficient distributed training, covering everything from algorithmic tricks like gradient compression to hardware innovations like in-network aggregation.

What Problem Does This Paper Address?

Traditional deep learning (DL) often happens on a single machine. However, modern foundation models and massive datasets require distributed DL—spreading the workload across hundreds or thousands of GPUs. In these large-scale environments, the "communication wall" is real: the time it takes to synchronize model weights and share data across nodes often exceeds the time spent on actual computation.

The authors identify three primary challenges in this space:

  1. Algorithmic Complexity: As the number of devices grows, traditional synchronization methods introduce massive overhead.
  2. Heterogeneity: Real-world clusters have varying hardware speeds, network bandwidths, and data distributions (especially in Federated Learning).
  3. Large Model Constraints: Models like LLMs are too big for a single GPU's memory, necessitating complex partitioning strategies.

Methods

The survey categorizes solutions into three distinct layers, as organized in the paper's roadmap:

Fig. 1. The organization of the survey
  • Algorithm Level: The paper reviews variants of Distributed Stochastic Gradient Descent (SGD), including asynchronous and "stale" synchronization methods that let fast workers keep going without waiting for "stragglers." It also dives deep into Data Compression, specifically Quantization (reducing the precision of gradients, e.g., to 1-bit) and Sparsification (only sending the most important gradients).
  • Framework Level: This focuses on Resource Allocation and Task Scheduling. The goal here is to maximize GPU utilization (which is often as low as 25-50% in production) by overlapping communication with computation and using "elastic" training that adjusts resources on the fly.
  • Infrastructure Level: The authors examine the physical "plumbing" of distributed DL, including high-speed interconnects like NVIDIA’s NVLink, programmable SmartNICs that can offload tasks from the CPU, and specialized network topologies like Torus or HammingMesh designed for all-to-all communication.

Datasets

As a comprehensive survey, this paper does not introduce a new dataset. Instead, it references performance benchmarks across a wide variety of standard DL tasks, including:

  • Computer Vision: ImageNet (for training speedups).
  • NLP: BERT and various LLM training logs (for scalability analysis).
  • Federated Learning: Non-IID (non-independent and identically distributed) datasets that simulate real-world edge device data.

Key Insights and Ideas

  • The "Straggler" Problem: In large clusters, the slowest node dictates the speed of the entire training run. Hybrid SGD methods that group workers by speed can mitigate this.
  • 3D Parallelism is King: For LLMs, the most effective approach is a combination of data, pipeline, and model parallelism. Parallelism modes of distributed DL
  • In-Network Aggregation (INA): Instead of nodes sending gradients to a central server, modern switches can actually perform the mathematical summation of gradients while the data is in transit, drastically reducing network traffic.
  • Communication-Compute Overlap: The "holy grail" of scheduling is to ensure that while the GPU is busy calculating the next layer, the network is busy sending the results of the previous one.
Parallelism modes of distributed DL

Key Results

The survey highlights several impressive quantitative gains from the literature:

  • Compression: Hybrid methods like CocktailSGD can achieve up to 117x compression on gradients, allowing LLMs to be trained even over relatively slow 500 Mbps networks with minimal slowdown.
  • Speedup: Decentralized SGD algorithms have been proven to achieve linear speedup relative to the number of workers, effectively bypassing the bottleneck of a central parameter server.
  • Utilization: Frameworks like AntMan and Gandiva can significantly boost cluster-wide GPU utilization by co-scheduling jobs and using fine-grained GPU sharing.

Limitations and Future Work

Despite the progress, the authors note several open challenges:

  • The Compression Trade-off: While compression saves bandwidth, the act of compressing and decompressing data adds its own computational overhead. Finding the "sweet spot" for different model architectures is still an art.
  • Fault Tolerance at Scale: In a cluster with 3,000+ GPUs, hardware failure is a statistical certainty. Current scheduling needs to become more resilient to node failures without requiring a full training restart.
  • Heterogeneity Profiling: We need better automated tools to profile dynamic network environments in real-time so that algorithms can adapt their synchronization frequency on the fly.

For researchers looking to enter the field, the authors suggest that the intersection of LLMs and communication-efficient hardware (like Silicon Photonics) is the next major frontier.

Summary updated

Abstract

With the rapid growth in the volume of data sets, models, and devices in the domain of deep learning, there is increasing attention on large-scale distributed deep learning. In contrast to traditional distributed deep learning, the large-scale scenario poses new challenges that include fault tolerance, scalability of algorithms and infrastructures, and heterogeneity in data sets, models, and resources. Due to intensive synchronization of models and sharing of data across GPUs and computing nodes during distributed training and inference processes, communication efficiency becomes the bottleneck for achieving high performance at a large scale. This article surveys the literature over the period of 2018-2023 on algorithms and technologies aimed at achieving efficient communication in large-scale distributed deep learning at various levels, including algorithms, frameworks, and infrastructures. Specifically, we first introduce efficient algorithms for model synchronization and communication data compression in the context of large-scale distributed training. Next, we introduce efficient strategies related to resource allocation and task scheduling for use in distributed training and inference. After that, we present the latest technologies pertaining to modern communication infrastructures used in distributed deep learning with a focus on examining the impact of the communication overhead in a large-scale and heterogeneous setting. Finally, we conduct a case study on the distributed training of large language models at a large scale to illustrate how to apply these technologies in real cases. This article aims to offer researchers a comprehensive understanding of the current landscape of large-scale distributed deep learning and to reveal promising future research directions toward communication-efficient solutions in this scope.
BibTeX
@misc{liang2024communication,
  author = {Feng Liang and Zhen Zhang and Haifeng Lu and Victor C. M. Leung and Yanyi Guo and Xiping Hu},
  title = {{Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey}},
  year = {2024},
  eprint = {2404.06114},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.06114},
}

Loading related content...

Figure