TL;DR
This paper systematically investigates the crucial tradeoff between model accuracy and training runtime in distributed deep learning. They introduce "Rudra," a framework for studying this, and propose novel synchronization protocols and a learning rate strategy that significantly improve efficiency without sacrificing accuracy, revealing that scaling out requires careful adjustments to mini-batch sizes.
What Problem Does This Paper Address?
Training large-scale deep neural networks (DNNs) has become computationally intensive, necessitating distributed training across many machines. While distributing computation can speed up training, it introduces complexities. Specifically, there's a lack of systematic understanding of how different distributed training strategies (like synchronization protocols, gradient staleness, and the number of workers) interact with DNN hyperparameters (like mini-batch size and learning rate) to affect both the final model accuracy and the overall training time. This paper aims to fill that gap by providing a principled study of these interdependencies.
Methods
The authors developed Rudra, a parameter server-based distributed computing framework designed for large-scale DNN training. Rudra allows for systematic study of various distributed training configurations. Key methodological contributions include:
- Vector Clock for Staleness Quantification: A technique to precisely measure the "staleness" of gradients, which is crucial for understanding asynchronous training.
- New Synchronization Protocols:
- n-softsync: A protocol where the parameter server waits for a minimum number of gradients (controlled by 'n') before updating weights, balancing staleness and communication overhead.
- Rudra-adv and Rudra-adv*: Architectures that use a parameter server group organized in a tree structure to reduce network traffic to the central parameter server and improve bandwidth utilization.
- Staleness-Aware Learning Rate Modulation: A strategy where the learning rate is dynamically adjusted based on the average gradient staleness (α = α₀/σ), helping to mitigate the negative effects of stale updates.
Datasets
The study utilized two standard image classification benchmarks:
- CIFAR10: Used for initial design space exploration and evaluating the impact of hyperparameter configurations.
- ImageNet: Used for evaluating the performance of the Rudra-adv and Rudra-adv* architectures on a larger, more complex dataset.
Key Insights and Ideas
- Asynchrony Isn't Always Better: Contrary to popular belief, purely asynchronous SGD can lead to complex interdependencies that negatively impact accuracy and runtime.
- Staleness is a Double-Edged Sword: While some staleness is inherent in asynchronous methods, excessive staleness degrades accuracy. The proposed n-softsync protocol and learning rate modulation help manage this.
- Scaling Out Requires Shrinking Mini-Batches: To maintain model accuracy as the number of learners (λ) increases, the mini-batch size per learner (μ) must be reduced. This suggests a fundamental limit to parallelism if mini-batch sizes are kept constant.
- The μλ Product Matters: For a given synchronization protocol (like hardsync or n-softsync), configurations that maintain a constant product of mini-batch size and number of learners (μλ) tend to yield similar test errors.
Key Results
- Improved Runtime: The Rudra-adv* architecture, using the n-softsync protocol, achieved significant speedups, reducing training time per epoch by up to 4.5x compared to a baseline hardsync configuration on ImageNet (125 min/epoch vs. 330 min/epoch).
- Accuracy Preservation: Despite speedups, the advanced configurations (Rudra-adv, Rudra-adv*) maintained comparable top-1 and top-5 validation errors to the baseline on ImageNet.
- Learning Rate Modulation Benefits: On CIFAR10, dividing the learning rate by the average staleness (α = α₀/σ) led to significantly lower test errors, especially in configurations with higher staleness. For example, a 30-softsync, λ=30 configuration that failed to converge with a constant learning rate achieved much lower error when the learning rate was modulated.
- Scalability Limits: On CIFAR10, the test error increased monotonically with the μλ product, indicating that simply adding more learners without adjusting mini-batch size will eventually lead to degraded accuracy.
Limitations and Future Work
- Theoretical Underpinning: The paper mentions that a separate paper (Zhang et al. [2017]) provides theoretical justification for the empirical results, suggesting that this work is primarily empirical.
- Optimization Algorithms: The authors point out that their findings highlight a need for designing optimization algorithms and learning strategies that perform well with larger batch sizes, as this is a key challenge for future work.
- Broader Hyperparameter Space: While systematic, the study focuses on specific protocols and hyperparameters. Exploring a wider range of optimization algorithms and network architectures in distributed settings could yield further insights.
Summary updated