Abstract
Distributed deep learning often faces a trade-off between the synchronization overhead of Synchronous SGD (SSGD) and the instability caused by gradient staleness in Asynchronous SGD (ASGD). This paper introduces a "staleness-aware" ASGD that modulates the learning rate based on the age of each gradient. By scaling the learning rate inversely with staleness ($\alpha = \alpha_0 / \tau$), the proposed method achieves near-linear speedup while maintaining model accuracy comparable to synchronous training. Experiments on CIFAR-10 and ImageNet demonstrate that this approach effectively mitigates the "asynchronous penalty," enabling efficient scaling to large numbers of learners.
Introduction
Training large-scale deep neural networks requires distributed computing to manage computational demands. The standard Synchronous SGD (SSGD) approach ensures stability by requiring all workers to synchronize at a "barrier" before updating the model. However, this leads to significant idle time as faster workers wait for the slowest "straggler."
Asynchronous SGD (ASGD) eliminates this barrier, allowing workers to update the central model immediately upon completing a computation. While faster, ASGD introduces staleness: gradients are often computed using an outdated version of the model. As the number of workers increases, staleness grows, which can destabilize training and lead to poor convergence or total divergence.
Methods
The authors propose two primary mechanisms to address the challenges of asynchronous training:
- The $n$-softsync Protocol: This protocol provides a flexible synchronization scheme that bounds staleness. Unlike SSGD (which waits for all workers) or traditional ASGD (which waits for none), the parameter server updates weights after receiving gradients from a specific subset of learners. This allows for fine-grained control over the distribution of gradient ages.
- Staleness-Aware Learning Rate: The core innovation is a per-gradient learning rate modulation. For each incoming gradient, the system calculates its staleness ($\tau$), defined as the difference between the current model timestamp and the timestamp of the weights used for the gradient calculation. The learning rate ($\alpha$) is adjusted as follows: $$\alpha_{stale} = \frac{\alpha_0}{\tau}$$ This linear penalty ensures that older, less reliable gradients have a smaller impact on the model update, providing a simple yet theoretically sound method for stabilizing training.
Results and Discussion
Convergence and Accuracy
The effectiveness of the staleness-aware modulation is most evident in training stability. In experiments on the CIFAR-10 dataset, traditional ASGD (Downpour-SGD) failed to converge as the number of learners increased, resulting in test error rates near 90%.
As shown in the figure above, applying the staleness-aware learning rate ($\alpha = \alpha_0 / \tau$) allows training to converge regardless of the degree of staleness. The final test error reached approximately 18%, matching the performance of both the single-learner baseline and the synchronous "Hardsync" approach. Similar results were observed on the ImageNet dataset using the AlexNet architecture, where the method achieved a top-1 validation error of ~43%, matching the high-accuracy SSGD baseline.
Scaling and Speedup
The proposed method enables significant performance gains without sacrificing accuracy. On an HPC cluster with 30 learners, the implementation achieved a 22x to 28x speedup compared to a single learner.
The scaling behavior is nearly linear, as illustrated in the speedup plots for both CIFAR-10 and ImageNet. On average, the staleness-aware ASGD approach was 50% faster than its synchronous counterpart, primarily due to the elimination of synchronization barriers and straggler effects.
Conclusion
The staleness-aware ASGD protocol offers a robust solution for distributed deep learning. By simply dividing the learning rate by the staleness value, the system gains "immunity" to the negative effects of delayed gradients. This approach provides the best of both worlds: the high throughput of asynchronous execution and the convergence stability of synchronous training.
Summary updated