LF-MMI Training of Bayesian and Gaussian Process Time Delay Neural Networks for Speech Recognition

Shoukang Hu, Xurong Xie, Shansong Liu, Max W. Y. Lam, Jianwei Yu, Xixin Wu, Xunying Liu, Helen Meng

Abstract

This paper explores improving the generalization of speech recognition models by replacing deterministic hidden activations in Time Delay Neural Networks (TDNNs) with probabilistic Bayesian and Gaussian Process (GP) activations. By accounting for parameter uncertainty during Lattice-Free Maximum Mutual Information (LF-MMI) training, the authors achieve consistent Word Error Rate (WER) reductions on the Switchboard dataset, particularly in data-constrained scenarios.

Introduction: The Challenge of Overfitting in ASR

In Automatic Speech Recognition (ASR), discriminative training criteria like LF-MMI are the industry standard for achieving high performance. However, these methods typically produce "deterministic" models—meaning they learn specific, fixed values for every weight in the network.

The core issue is overfitting. When training data is limited, these deterministic models tend to "memorize" the noise or specific quirks of the training set rather than learning generalizable patterns of human speech. While standard regularization techniques (like dropout or $L_2$ norm) help, they don't explicitly model the uncertainty inherent in the model's parameters. The authors aim to bridge this gap by introducing a probabilistic framework into the LF-MMI training pipeline.

Proposed Methods: B-TDNN and GP-TDNN

The authors propose two primary modifications to the standard TDNN architecture to incorporate uncertainty:

  1. Bayesian TDNN (B-TDNN): Instead of fixed weights, each weight is treated as a random variable drawn from a posterior distribution. During the forward pass, the hidden node output is calculated by marginalizing (averaging) over these distributions.
  2. Gaussian Process TDNN (GP-TDNN): This goes a step further by treating the basis coefficients of the activation functions as uncertain variables. This allows the model more flexibility in learning the actual shape of the activation function while accounting for uncertainty.
Architecture overview of TDNN, B-TDNN, and GP-TDNN models showing the integration of probabilistic activations.

Technical Innovations

  • Variational Inference: To make the complex integrations required by Bayesian learning tractable, the authors use a Variational Lower Bound approach. This approximates the true posterior with a simpler Gaussian distribution.
  • Monte Carlo Sampling: During training, the authors use Monte Carlo sampling to estimate gradients. Interestingly, they found that taking just one sample per training step was sufficient to train the models effectively.
  • LF-MMI Integration: They successfully integrated these probabilistic layers into the "chain" model framework (LF-MMI), which involves a complex sequence-level objective function rather than simple frame-level cross-entropy.

Experimental Setup

The models were evaluated on the Switchboard conversational telephone speech task, a standard benchmark in the ASR community:

  • 75-hour subset: Used to test the models in a limited-data environment where overfitting is most prevalent.
  • 300-hour full set: Used to evaluate the scalability and performance of the techniques on a larger, more standard training set.
  • Evaluation Sets: Performance was measured across several standard test sets, including HUB5 2000, RT03S, and RT02.

Key Results and Insights

The primary insight is that treating weights as distributions acts as a powerful, mathematically grounded form of regularization. It forces the network to find solutions that are robust across a range of likely parameter values.

Performance on Data-Constrained Tasks

On the 75-hour subset, the benefits of the probabilistic approach were most pronounced. As shown in the results table below, both B-TDNN and GP-TDNN variants outperformed the baseline TDNN across all test sets.

Word Error Rate (WER) results on the 75-hour Switchboard subset across different test sets.
  • 75-Hour Task: On the CHM subset of Hub5 '00, the B-TDNN reduced the absolute WER by 0.7% over the baseline. The GP-TDNN provided an even larger boost, reaching a 1.6% absolute reduction compared to the deterministic baseline.
  • 300-Hour Task: On the full dataset, the B-TDNN continued to show consistent improvements (e.g., a 0.4% absolute reduction on the SWB1 subset).
  • GP Flexibility: The GP-TDNN variants showed that adding uncertainty to the activation function itself provides an extra layer of protection against overfitting, especially when the training set is small. However, on the larger 300h dataset, the GP-TDNNs performed similarly to the B-TDNNs, suggesting diminishing returns for GP flexibility as data volume increases.

Conclusion and Future Directions

This research demonstrates that Bayesian neural networks can be efficiently trained within the LF-MMI framework using a "sample once" strategy. While these models introduce some additional complexity and hyperparameters, the gains in generalization—particularly in low-resource settings—are significant.

Future work may involve exploring the application of these Bayesian LF-MMI techniques to other architectures like LSTMs or Transformers, and investigating methods to better leverage GP activations when data is more abundant to overcome the observed saturation on larger datasets.

Summary updated

Abstract

Discriminative training techniques define state-of-the-art performance for deep neural networks (DNNs) based speech recognition systems across a wide range of tasks. Conventional discriminative training methods produce deterministic DNN parameter estimates. They are inherently prone to overfitting, leading to poor generalization when given limited training data. In order to address this issue, this paper investigates the use of Bayesian learning and Gaussian Process (GP) based hidden activations to replace the deterministic parameter estimates of standard lattice-free maximum mutual information (LF-MMI) criterion trained time delay neural network (TDNN) acoustic models. Experiments conducted on the Switchboard conversational telephone speech recognition tasks suggest the proposed technique consistently outperforms the baseline LF-MMI trained TDNN systems using fixed parameter hidden activations.
BibTeX
@inproceedings{hu2019lf,
  author = {Shoukang Hu and Xurong Xie and Shansong Liu and Max W. Y. Lam and Jianwei Yu and Xixin Wu and Xunying Liu and Helen Meng},
  title = {{LF-MMI Training of Bayesian and Gaussian Process Time Delay Neural Networks for Speech Recognition}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2379},
}
Figure