Layered SGD: A Decentralized and Synchronous SGD Algorithm for Scalable Deep Neural Network Training

Kwangmin Yu, Thomas Flynn, Shinjae Yoo, Nicholas D'ImperioarXiv 2019 · OpenReview

Abstract

Stochastic Gradient Descent (SGD) is the most popular algorithm for training deep neural networks (DNNs). As larger networks and datasets cause longer training times, training on distributed systems is common and distributed SGD variants, mainly asynchronous and synchronous SGD, are widely used. Asynchronous SGD is communication efficient but suffers from accuracy degradation due to delayed parameter updating. Synchronous SGD becomes communication intensive when the number of nodes increases regardless of its advantage. To address these issues, we introduce Layered SGD (LSGD), a new decentralized synchronous SGD algorithm. LSGD partitions computing resources into subgroups that each contain a communication layer (communicator) and a computation layer (worker). Each subgroup has centralized communication for parameter updates while communication between subgroups is handled by communicators. As a result, communication time is overlapped with I/O latency of workers. The efficiency of the algorithm is tested by training a deep network on the ImageNet classification task.

Introduction

Training deep neural networks on massive datasets requires distributed computing, but scaling Stochastic Gradient Descent (SGD) across multiple nodes introduces a significant trade-off. Currently, researchers generally choose between two paradigms: Synchronous SGD, which guarantees mathematically unbiased updates but suffers from communication bottlenecks, and Asynchronous SGD, which scales well but degrades accuracy due to "stale gradients."

Layered SGD (LSGD) is designed to bridge this gap, offering a synchronous, mathematically exact SGD algorithm that scales as efficiently as decentralized or asynchronous methods.

Methodology

To solve the communication bottleneck, LSGD decouples communication and computation tasks, hiding network latency behind standard I/O operations.

Hierarchical Topology

Instead of a flat network where every GPU communicates with every other GPU, LSGD partitions compute resources into subgroups (e.g., a single physical server). Each subgroup contains several "workers" (GPUs) and one "communicator" (a CPU core).

Comparison of distributed SGD topologies: Centralized (left), Ring (center), and the proposed Layered SGD (right).

Two-Phase Communication and I/O Overlap

The communication happens in two phases:

  1. Local Reduce: Workers compute gradients and aggregate them locally to their node's communicator.
  2. Global Allreduce: The communicators perform the inter-node Allreduce operation across the network.

The true novelty of LSGD lies in reordering the training loop. While the CPUs are busy syncing gradients globally, the GPUs immediately start loading the next mini-batch of training data from disk. If the I/O time is longer than the communication time, the network overhead is effectively hidden.

Detailed view of the LSGD topology showing subgroups with four workers per communicator.

Experimental Results

The algorithm was benchmarked using a ResNet-50 model on the ImageNet (ILSVRC2012) dataset.

  • Scalability: LSGD demonstrated near-perfect linear scalability. At 256 GPUs, LSGD maintained a 93.1% scaling efficiency, significantly outperforming conventional distributed SGD (CSGD), which dropped to 63.8%.
  • Throughput: Throughput scales linearly as nodes are added, avoiding the diminishing returns seen in standard Allreduce-based methods.
  • Accuracy: Because LSGD is mathematically equivalent to synchronous SGD, it achieves the same validation accuracy (~73% Top-1 on ImageNet) without the staleness issues of asynchronous methods.

Conclusion

LSGD proves that network latency can be mitigated by cleverly scheduling operations and offloading communication to CPUs. While the algorithm scales exceptionally well, extremely large batch sizes (e.g., 16,384 at 256 workers) can still lead to generalization challenges. Future work will explore integrating Layer-wise Adaptive Rate Scaling (LARS) and testing on larger supercomputing clusters to further enhance performance and accuracy.

Summary updated

Abstract

Stochastic Gradient Descent (SGD) is the most popular algorithm for training deep neural networks (DNNs). As larger networks and datasets cause longer training times, training on distributed systems is common and distributed SGD variants, mainly asynchronous and synchronous SGD, are widely used. Asynchronous SGD is communication efficient but suffers from accuracy degradation due to delayed parameter updating. Synchronous SGD becomes communication intensive when the number of nodes increases regardless of its advantage. To address these issues, we introduce Layered SGD (LSGD), a new decentralized synchronous SGD algorithm. LSGD partitions computing resources into subgroups that each contain a communication layer (communicator) and a computation layer (worker). Each subgroup has centralized communication for parameter updates while communication between subgroups is handled by communicators. As a result, communication time is overlapped with I/O latency of workers. The efficiency of the algorithm is tested by training a deep network on the ImageNet classification task.
BibTeX
@misc{yu2019layered,
  author = {Kwangmin Yu and Thomas Flynn and Shinjae Yoo and Nicholas D'Imperio},
  title = {{Layered SGD: A Decentralized and Synchronous SGD Algorithm for Scalable Deep Neural Network Training}},
  year = {2019},
  eprint = {1906.05936},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.05936},
}
Figure