Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study

Suyog Gupta, Wei Zhang, Fei WangIJCAI 2017 · DOI

TL;DR

This paper systematically investigates the crucial tradeoff between model accuracy and training runtime in distributed deep learning. They introduce "Rudra," a framework for studying this, and propose novel synchronization protocols and a learning rate strategy that significantly improve efficiency without sacrificing accuracy, revealing that scaling out requires careful adjustments to mini-batch sizes.

What Problem Does This Paper Address?

Training large-scale deep neural networks (DNNs) has become computationally intensive, necessitating distributed training across many machines. While distributing computation can speed up training, it introduces complexities. Specifically, there's a lack of systematic understanding of how different distributed training strategies (like synchronization protocols, gradient staleness, and the number of workers) interact with DNN hyperparameters (like mini-batch size and learning rate) to affect both the final model accuracy and the overall training time. This paper aims to fill that gap by providing a principled study of these interdependencies.

Methods

The authors developed Rudra, a parameter server-based distributed computing framework designed for large-scale DNN training. Rudra allows for systematic study of various distributed training configurations. Key methodological contributions include:

  • Vector Clock for Staleness Quantification: A technique to precisely measure the "staleness" of gradients, which is crucial for understanding asynchronous training.
  • New Synchronization Protocols:
    • n-softsync: A protocol where the parameter server waits for a minimum number of gradients (controlled by 'n') before updating weights, balancing staleness and communication overhead.
  • Rudra-adv and Rudra-adv*: Architectures that use a parameter server group organized in a tree structure to reduce network traffic to the central parameter server and improve bandwidth utilization.
  • Staleness-Aware Learning Rate Modulation: A strategy where the learning rate is dynamically adjusted based on the average gradient staleness (α = α₀/σ), helping to mitigate the negative effects of stale updates.

Datasets

The study utilized two standard image classification benchmarks:

  • CIFAR10: Used for initial design space exploration and evaluating the impact of hyperparameter configurations.
  • ImageNet: Used for evaluating the performance of the Rudra-adv and Rudra-adv* architectures on a larger, more complex dataset.

Key Insights and Ideas

  • Asynchrony Isn't Always Better: Contrary to popular belief, purely asynchronous SGD can lead to complex interdependencies that negatively impact accuracy and runtime.
  • Staleness is a Double-Edged Sword: While some staleness is inherent in asynchronous methods, excessive staleness degrades accuracy. The proposed n-softsync protocol and learning rate modulation help manage this.
  • Scaling Out Requires Shrinking Mini-Batches: To maintain model accuracy as the number of learners (λ) increases, the mini-batch size per learner (μ) must be reduced. This suggests a fundamental limit to parallelism if mini-batch sizes are kept constant.
  • The μλ Product Matters: For a given synchronization protocol (like hardsync or n-softsync), configurations that maintain a constant product of mini-batch size and number of learners (μλ) tend to yield similar test errors.

Key Results

  • Improved Runtime: The Rudra-adv* architecture, using the n-softsync protocol, achieved significant speedups, reducing training time per epoch by up to 4.5x compared to a baseline hardsync configuration on ImageNet (125 min/epoch vs. 330 min/epoch).
  • Accuracy Preservation: Despite speedups, the advanced configurations (Rudra-adv, Rudra-adv*) maintained comparable top-1 and top-5 validation errors to the baseline on ImageNet.
  • Learning Rate Modulation Benefits: On CIFAR10, dividing the learning rate by the average staleness (α = α₀/σ) led to significantly lower test errors, especially in configurations with higher staleness. For example, a 30-softsync, λ=30 configuration that failed to converge with a constant learning rate achieved much lower error when the learning rate was modulated.
  • Scalability Limits: On CIFAR10, the test error increased monotonically with the μλ product, indicating that simply adding more learners without adjusting mini-batch size will eventually lead to degraded accuracy.

Limitations and Future Work

  • Theoretical Underpinning: The paper mentions that a separate paper (Zhang et al. [2017]) provides theoretical justification for the empirical results, suggesting that this work is primarily empirical.
  • Optimization Algorithms: The authors point out that their findings highlight a need for designing optimization algorithms and learning strategies that perform well with larger batch sizes, as this is a key challenge for future work.
  • Broader Hyperparameter Space: While systematic, the study focuses on specific protocols and hyperparameters. Exploring a wider range of optimization algorithms and network architectures in distributed settings could yield further insights.

Summary updated

BibTeX
@inproceedings{gupta2017model,
  author = {Suyog Gupta and Wei Zhang and Fei Wang},
  title = {{Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/681},
}
Figure