Scalable Minimum Bayes Risk Training of Deep Neural Network Acoustic Models Using Distributed Hessian-free Optimization

Brian Kingsbury, Tara N. Sainath, Hagen SoltauInterspeech 2012 · OpenReview

Abstract

This paper presents a distributed training framework for Deep Neural Network (DNN) acoustic models that utilizes sequence-discriminative criteria (state-level Minimum Bayes Risk, sMBR) and second-order Hessian-free (HF) optimization. By parallelizing curvature calculations in a master-worker architecture, the approach achieves significant speedups—up to 5.5x over standard stochastic gradient descent (SGD)—and improves word error rates (WER) across large-scale speech recognition tasks including Switchboard and noisy Levantine Arabic.

Introduction

By 2012, Deep Neural Networks (DNNs) had significantly improved speech recognition performance compared to traditional Gaussian Mixture Models (GMMs). However, training these models efficiently on large datasets remained a challenge. Two primary bottlenecks were identified: the use of frame-level cross-entropy (CE) objective functions, which ignore the sequential nature of speech, and the reliance on Stochastic Gradient Descent (SGD), which is difficult to parallelize effectively across distributed systems.

While sequence-discriminative criteria like state-level Minimum Bayes Risk (sMBR) are theoretically superior, they are computationally intensive. This paper addresses these challenges by introducing a scalable framework that combines sMBR with Hessian-free (HF) optimization in a distributed environment.

Methodology

The core of the proposed approach is the application of Hessian-free (HF) optimization within a distributed master-worker architecture.

Second-Order Optimization

Unlike first-order methods like SGD that only consider the gradient (slope), HF optimization utilizes the Hessian matrix (curvature) of the loss function. This allows for more informed update steps, leading to faster convergence in fewer iterations. To handle the computational complexity of the Hessian in high-dimensional parameter spaces, the authors employ the Conjugate Gradient (CG) algorithm. This "Hessian-free" trick allows for the calculation of matrix-vector products without explicitly forming the full Hessian matrix.

Distributed Architecture

To scale training to massive datasets, the authors implemented a data-parallel master-worker setup. Multiple worker nodes compute gradients and curvature products in parallel across different data chunks, while a master node coordinates the optimization steps. This architecture is particularly well-suited for HF optimization, which benefits from the large batch sizes that distributed clusters can provide.

Training iterations vs. per-sample loss and WER, showing convergence dynamics and the impact of acoustic weighting.

Experimental Setup

The framework was evaluated on three distinct tasks:

  1. English Broadcast News (50 hours): Benchmarking speed and basic performance.
  2. Switchboard (300 hours): A large-scale conversational speech task.
  3. Noisy Levantine Arabic (249 hours): A challenging task involving highly distorted radio transmissions.

Results

The experimental results demonstrate that the HF-sMBR approach provides substantial gains in both training efficiency and recognition accuracy.

Efficiency Gains

On the 50-hour Broadcast News task, the distributed HF-sMBR algorithm achieved a 5.5x speedup over the standard SGD-sMBR implementation, reducing total training time from 101 hours to just 18 hours.

Accuracy Improvements

The use of sequence-level training consistently outperformed frame-level baselines. On the Switchboard task, the HF-sMBR model reached a 16.4% WER, representing a 13% relative improvement over the cross-entropy baseline. Similar relative improvements (7–13%) were observed in the noisy Arabic task.

Comparison of Word Error Rates (WER) across different model types and optimization strategies on Broadcast News test sets.

As shown in the table above, the HF-sMBR optimization strategy consistently yields lower WER compared to both cross-entropy and SGD-based sMBR training.

Conclusion

This work demonstrates that second-order Hessian-free optimization can be effectively scaled to large-scale speech recognition tasks using distributed computing. By enabling efficient sequence-discriminative training, the proposed framework offers a significant advantage over traditional SGD-based methods in both speed and final model accuracy. Future work may explore the use of preconditioning to further accelerate the Conjugate Gradient steps and adapt the framework for multi-GPU environments.

Summary updated

BibTeX
@inproceedings{kingsbury2012scalable,
  author = {Brian Kingsbury and Tara N. Sainath and Hagen Soltau},
  title = {{Scalable Minimum Bayes Risk Training of Deep Neural Network Acoustic Models Using Distributed Hessian-free Optimization}},
  booktitle = {Interspeech},
  year = {2012},
  url = {https://openreview.net/forum?id=r0gCDZ0RIxc},
}
Figure