Pipelined Back-Propagation for Context-Dependent Deep Neural Networks

Xie Chen, Adam Eversole, Gang Li, Dong Yu, Frank Seide

In the early 2010s, the field of speech recognition underwent a paradigm shift with the introduction of Context-Dependent Deep Neural Network HMMs (CD-DNN-HMMs). While these models significantly outperformed traditional Gaussian Mixture Models (GMMs), they introduced a daunting practical challenge: the immense computational cost of training. This 2012 paper from Microsoft Research presents a novel parallelization framework designed to slash training times without compromising the accuracy of the resulting models.

The Training Bottleneck

By 2012, CD-DNN-HMMs had established themselves as the state-of-the-art for speech-to-text tasks. However, their training process was notoriously slow. For instance, training a large-scale model on a single GPU could take nearly two months—approximately 59 to 68 days depending on the configuration.

Standard parallelization techniques, such as Map-Reduce, which were effective for GMMs, proved unsuitable for DNNs. DNN training relies on Stochastic Gradient Descent (SGD) with small minibatches—typically a few hundred frames of audio. Updating the model after each minibatch requires frequent and massive data transfers. In a multi-server environment, these transfers are limited by network bandwidth; even within a single server, the PCIe bus can become a bottleneck. The researchers sought a method to leverage multiple GPUs within a single workstation effectively.

Methodology: Pipelined Back-Propagation and Model Striping

The authors proposed a two-pronged approach to overcome these hardware limitations:

1. Pipelined Back-Propagation

Instead of partitioning the training data (data parallelism), the authors partitioned the network itself across multiple GPUs (model parallelism). In a pipelined setup, different GPUs handle different layers of the neural network. For example, in a 2-GPU configuration, the first GPU might process the lower layers while the second GPU processes the upper layers.

This architecture allows for simultaneous computation: while the second GPU calculates gradients for the current minibatch, the first GPU can begin the forward pass for the subsequent minibatch. Although this introduces a "pipeline delay"—meaning weight updates are based on slightly "stale" gradients—the authors demonstrated that this delay is manageable. Since standard minibatch training already involves a form of delayed update (the model is only updated after the entire batch is processed), as long as the pipeline delay remains within the magnitude of typical minibatch sizes, convergence and accuracy are preserved.

2. Model Striping

A specific challenge in speech recognition is the "top-heavy" nature of the networks. The output layer, which represents thousands of "senones" (tied triphone states), is often significantly larger than the hidden layers. This creates a computational imbalance where the GPU handling the final layers becomes a bottleneck. To address this, the authors implemented "model striping," where the large output layer is partitioned vertically across multiple GPUs, balancing the workload and maximizing throughput.

Experimental Results

The proposed methods were evaluated using the Switchboard-I (309 hours) and Fisher (2000 hours) corpora. The results confirmed that the pipelined approach provides substantial speedups with zero degradation in Word Error Rate (WER).

Table 5: Word Error Rate (WER) and training time comparison. The DNN models show a dramatic reduction in WER compared to GMM baselines, though at the cost of significantly higher training times.

As shown in the performance data, the transition from GMMs to DNNs provided a relative WER reduction of over 30% on the Switchboard task. However, the training time for the largest models reached nearly 60 days on a single GPU baseline.

Table 6: Training time (in days) for various parallelization methods and minibatch sizes. The combination of pipelining and model striping reduced training time from 59 days to 18 days.

By utilizing a 4-GPU configuration with pipelined training and a striped top layer, the researchers achieved a 3.3x end-to-end speedup. This reduced the training time for the most complex models from 59 days to just 18 days, representing a parallelization efficiency of approximately 82%.

Conclusion

The introduction of pipelined back-propagation and model striping provided a critical solution to the training bottleneck of early deep learning models in speech recognition. By carefully managing the trade-off between update "staleness" and hardware throughput, the authors enabled the practical training of large-scale DNNs on massive datasets like the 2000-hour Fisher corpus. This work laid the groundwork for the rapid scaling of neural networks that would follow in the subsequent decade.

Summary updated

BibTeX
@inproceedings{chen2012pipelined,
  author = {Xie Chen and Adam Eversole and Gang Li and Dong Yu and Frank Seide},
  title = {{Pipelined Back-Propagation for Context-Dependent Deep Neural Networks}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-7},
}
Figure