Abstract
In this paper, we propose and investigate a variety of distributed deep learning strategies for automatic speech recognition (ASR) and evaluate them with a state-of-the-art Long short-term memory (LSTM) acoustic model on the 2000-hour Switchboard (SWB2000). We first investigate proper hyper-parameters to enable training with sufficiently large batch sizes without impairing model accuracy. We then implement various distributed strategies, including Synchronous (SYNC), Asynchronous Decentralized Parallel SGD (ADPSGD), and a HYBRID approach, to study their runtime/accuracy trade-off. We demonstrate that the LSTM model can be trained using ADPSGD in 14 hours with 16 NVIDIA P100 GPUs to reach a 7.6% WER on the Hub5-2000 Switchboard (SWB) test set. Furthermore, using the HYBRID strategy, training is completed in 11.5 hours with 32 NVIDIA V100 GPUs without loss in accuracy.
Introduction
Training deep learning models for Automatic Speech Recognition (ASR) presents unique scaling challenges compared to Computer Vision (CV). While CV models like ResNet can be trained with massive batch sizes (8192+), ASR models—specifically LSTMs—have traditionally been limited to small batches (around 256) to maintain accuracy. ASR tasks also exhibit a low computation-to-communication ratio; the Switchboard-LSTM task is approximately five times more challenging to parallelize than ImageNet-ResNet. In standard synchronous training, a single "straggler" (a slow GPU or network node) can bottleneck the entire cluster, leading to significant resource underutilization.
Methodology
Large Batch Training
To enable more parallel learners, the researchers increased the batch size from 256 to 2560. To mitigate the accuracy degradation typically associated with large batches, a linear learning rate warmup was employed, scaling from 0.1 to 1.0 over the first 10 epochs, followed by a specific annealing schedule. As shown in the loss curves, this approach allows the larger batch size to achieve convergence parity with the baseline.
Distributed Strategies
The study compared three primary architectural strategies:
- Synchronous (SYNC): All learners calculate gradients and wait for a global update via a central parameter server or AllReduce operation.
- Asynchronous Decentralized Parallel SGD (ADPSGD): Learners are organized in a ring topology. Each learner updates its own weights and then averages them with a neighbor asynchronously. This removes the central bottleneck and the "straggler problem."
- HYBRID: A combination using optimized "AllReduce" operations to minimize weight discrepancy while allowing for communication/computation overlap.
Results and Discussion
The models were evaluated on the NIST Hub5 2000 test sets (Switchboard and CallHome). Using the HYBRID strategy on 32 NVIDIA V100 GPUs, training was completed in just 11.5 hours. The models achieved a 7.6% Word Error Rate (WER) on SWB and 13.1% on CH, matching the accuracy of a single-GPU baseline.
A key finding is ADPSGD's immunity to stragglers. In controlled tests where one GPU was slowed by 100x, ADPSGD maintained nearly its full speed-up, whereas synchronous methods were severely throttled. This decentralization, moving from a central Parameter Server to a ring-based peer-to-peer averaging system, effectively removes the primary communication bottleneck.
Conclusion
This work demonstrates that state-of-the-art ASR models can be trained in under 12 hours by scaling batch sizes and utilizing decentralized asynchronous strategies. Future work includes exploring "wait-free" implementations to further decouple communication and computation, as well as investigating mixed-precision training (FP16) for additional acceleration.
Summary updated