Distributed Deep Learning Strategies For Automatic Speech Recognition

Wei Zhang, Xiaodong Cui, Ulrich Finkler, Brian Kingsbury, George Saon, David S. Kung, Michael Picheny

Abstract

In this paper, we propose and investigate a variety of distributed deep learning strategies for automatic speech recognition (ASR) and evaluate them with a state-of-the-art Long short-term memory (LSTM) acoustic model on the 2000-hour Switchboard (SWB2000). We first investigate proper hyper-parameters to enable training with sufficiently large batch sizes without impairing model accuracy. We then implement various distributed strategies, including Synchronous (SYNC), Asynchronous Decentralized Parallel SGD (ADPSGD), and a HYBRID approach, to study their runtime/accuracy trade-off. We demonstrate that the LSTM model can be trained using ADPSGD in 14 hours with 16 NVIDIA P100 GPUs to reach a 7.6% WER on the Hub5-2000 Switchboard (SWB) test set. Furthermore, using the HYBRID strategy, training is completed in 11.5 hours with 32 NVIDIA V100 GPUs without loss in accuracy.

Introduction

Training deep learning models for Automatic Speech Recognition (ASR) presents unique scaling challenges compared to Computer Vision (CV). While CV models like ResNet can be trained with massive batch sizes (8192+), ASR models—specifically LSTMs—have traditionally been limited to small batches (around 256) to maintain accuracy. ASR tasks also exhibit a low computation-to-communication ratio; the Switchboard-LSTM task is approximately five times more challenging to parallelize than ImageNet-ResNet. In standard synchronous training, a single "straggler" (a slow GPU or network node) can bottleneck the entire cluster, leading to significant resource underutilization.

Methodology

Large Batch Training

To enable more parallel learners, the researchers increased the batch size from 256 to 2560. To mitigate the accuracy degradation typically associated with large batches, a linear learning rate warmup was employed, scaling from 0.1 to 1.0 over the first 10 epochs, followed by a specific annealing schedule. As shown in the loss curves, this approach allows the larger batch size to achieve convergence parity with the baseline.

Held-out loss comparison between standard (256) and large (2560) batch sizes, demonstrating convergence parity.

Distributed Strategies

The study compared three primary architectural strategies:

  1. Synchronous (SYNC): All learners calculate gradients and wait for a global update via a central parameter server or AllReduce operation.
  2. Asynchronous Decentralized Parallel SGD (ADPSGD): Learners are organized in a ring topology. Each learner updates its own weights and then averages them with a neighbor asynchronously. This removes the central bottleneck and the "straggler problem."
  3. HYBRID: A combination using optimized "AllReduce" operations to minimize weight discrepancy while allowing for communication/computation overlap.
Mechanism of Asynchronous Decentralized Parallel SGD (ADPSGD) showing overlapping computation and communication between nodes.

Results and Discussion

The models were evaluated on the NIST Hub5 2000 test sets (Switchboard and CallHome). Using the HYBRID strategy on 32 NVIDIA V100 GPUs, training was completed in just 11.5 hours. The models achieved a 7.6% Word Error Rate (WER) on SWB and 13.1% on CH, matching the accuracy of a single-GPU baseline.

A key finding is ADPSGD's immunity to stragglers. In controlled tests where one GPU was slowed by 100x, ADPSGD maintained nearly its full speed-up, whereas synchronous methods were severely throttled. This decentralization, moving from a central Parameter Server to a ring-based peer-to-peer averaging system, effectively removes the primary communication bottleneck.

Conclusion

This work demonstrates that state-of-the-art ASR models can be trained in under 12 hours by scaling batch sizes and utilizing decentralized asynchronous strategies. Future work includes exploring "wait-free" implementations to further decouple communication and computation, as well as investigating mixed-precision training (FP16) for additional acceleration.

Summary updated

Abstract

In this paper, we propose and investigate a variety of distributed deep learning strategies for automatic speech recognition (ASR) and evaluate them with a state-of-the-art Long short-term memory (LSTM) acoustic model on the 2000-hour Switchboard (SWB2000), which is one of the most widely used datasets for ASR performance benchmark. We first investigate what are the proper hyper-parameters (e.g., learning rate) to enable the training with sufficiently large batch size without impairing the model accuracy. We then implement various distributed strategies, including Synchronous (SYNC), Asynchronous Decentralized Parallel SGD (ADPSGD) and the hybrid of the two HYBRID, to study their runtime/accuracy trade-off. We show that we can train the LSTM model using ADPSGD in 14 hours with 16 NVIDIA P100 GPUs to reach a 7.6% WER on the Hub5- 2000 Switchboard (SWB) test set and a 13.1% WER on the CallHome (CH) test set. Furthermore, we can train the model using HYBRID in 11.5 hours with 32 NVIDIA V100 GPUs without loss in accuracy.
BibTeX
@inproceedings{zhang2019distributed,
  author = {Wei Zhang and Xiaodong Cui and Ulrich Finkler and Brian Kingsbury and George Saon and David S. Kung and Michael Picheny},
  title = {{Distributed Deep Learning Strategies For Automatic Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=BSgwPZA0Il5},
}
Figure