Unbiased Semi-Supervised LF-MMI Training Using Dropout

Sibo Tong, Apoorv Vyas, Philip N. Garner, Hervé BourlardInterspeech 2019 · OpenReview

Abstract

The lattice-free MMI objective (LF-MMI) with finite-state transducer (FST) supervision lattice has been used in semi-supervised training of state-of-the-art neural network acoustic models for automatic speech recognition (ASR). However, the FST based supervision lattice does not sample from the posterior predictive distribution of word-sequences but only contains the decoding hypotheses corresponding to the Maximum Likelihood estimate of weights, so that the training might be biased towards incorrect hypotheses in the supervision lattice even if the best path is perfectly correct. In this paper, we propose a novel framework which uses Dropout at the test time to sample from the posterior predictive distribution of word-sequences to produce unbiased supervision lattices for semi-supervised training. We investigate the dropout sampling from both the acoustic model and the language model to generate supervision. Results on Fisher English show that the proposed approach achieves WER recovery of ~51.6% over regular semi-supervised LF-MMI training.

Introduction

Training high-performance Automatic Speech Recognition (ASR) systems typically requires thousands of hours of transcribed audio, which is both time-consuming and expensive to acquire. Semi-supervised learning (SSL) offers a solution by leveraging small amounts of labeled data to "seed" a model, which then transcribes a much larger pool of unlabeled data.

In state-of-the-art systems using Lattice-Free Maximum Mutual Information (LF-MMI) training, researchers often use a "supervision lattice"—a graph representing multiple possible word sequences—to provide training signals. However, standard lattices generated from a single point estimate of model weights are often "biased." They may include many incorrect paths that the model erroneously treats as likely. If the seed model is biased, it propagates its errors to the new model, limiting the effectiveness of SSL.

Proposed Method: Dropout Sampling for Unbiased Lattices

To address this bias, we propose using Monte Carlo Dropout during the inference stage to represent model uncertainty. This approach treats Dropout not just as a training regularizer, but as a tool for approximate Bayesian inference, allowing the model to "know what it doesn't know."

Proposed framework for semi-supervised LF-MMI training using dropout sampling.

The framework consists of several key steps:

  1. Acoustic Model (AM) Dropout Sampling: Unlike standard decoding where Dropout is disabled, we keep it active. The same audio input is passed through the neural network $N$ times. Due to the random "dropping" of neurons, the model produces $N$ slightly different outputs, effectively sampling from its posterior predictive distribution.
  2. Lattice Composition: These $N$ hypotheses are pruned and combined into a single "unbiased" supervision lattice. In regions where the model is confident, the samples will be nearly identical. Where it is uncertain, the samples will vary, naturally highlighting alternative paths.
  3. Language Model (LM) Sampling: We extend this concept to the Language Model. By using a neural network-based LM with Dropout enabled during the re-scoring phase, we capture linguistic uncertainty alongside acoustic uncertainty.
  4. Training: The final ASR model is trained using these combined, unbiased lattices as the ground truth for the unlabeled data.

Experimental Setup

We evaluated our approach using the Fisher English corpus. A 50-hour subset served as the supervised "seed" data, while a 250-hour subset was used for semi-supervised training. Evaluation was performed on standard Kaldi recipe development and test sets (approximately 5 hours each). Transcripts from the remaining 1250 hours of the Fisher corpus were used to train the Language Models.

Key Results and Insights

The effectiveness of the proposed method is measured by Word Error Rate Recovery (WRR), which quantifies how much of the performance gap between the seed model and an "Oracle" model (trained on perfect transcripts) is closed.

WER vs. number of decoding times for acoustic model sampling.

As shown in the figure above, the performance gains from AM sampling are significant compared to the regular approach. We observed that the benefits of sampling begin to saturate at approximately $N=20$ decodings, providing a practical heuristic for implementation.

Key performance metrics include:

  • Regular Semi-Supervised: Achieved a WRR of 53.7%.
  • Proposed (AM Sampling): Achieved a WRR of 76.1%.
  • Proposed (AM + LM Sampling): Achieved the best result with a WRR of 77.6%.
  • Final WER: On the test set, our method reached a WER of 18.2%, a notable improvement over the 19.2% achieved by standard SSL and the 20.9% of the initial seed model.

While sampling from both the AM and LM provides the best results, the majority of the performance gains are attributable to AM sampling.

Conclusion and Future Work

This work demonstrates that using Dropout at inference time to generate unbiased supervision lattices significantly improves semi-supervised LF-MMI training. By capturing model uncertainty, we reduce the propagation of errors from seed models.

The primary limitation is the increased computational cost of decoding unlabeled data multiple times. Future research will explore more efficient uncertainty estimation techniques, such as Negative Conditional Entropy (NCE), and investigate applying these principles to end-to-end ASR architectures like Transformers.

Summary updated

Abstract

The lattice-free MMI objective (LF-MMI) with finite-state transducer (FST) supervision lattice has been used in semi-supervised training of state-of-the-art neural network acoustic models for automatic speech recognition (ASR). However, the FST based supervision lattice does not sample from the posterior predictive distribution of word-sequences but only contains the decoding hypotheses corresponding to the Maximum Likelihood estimate of weights, so that the training might be biased towards incorrect hypotheses in the supervision lattice even if the best path is perfectly correct. In this paper, we propose a novel framework which uses Dropout at the test time to sample from the posterior predictive distribution of word-sequences to produce unbiased supervision lattices for semi-supervised training. We investigate the dropout sampling from both the acoustic model and the language model to generate supervision. Results on Fisher English show that the proposed approach achieves WER recovery of ~51.6% over regular semi-supervised LF-MMI training.
BibTeX
@inproceedings{tong2019unbiased,
  author = {Sibo Tong and Apoorv Vyas and Philip N. Garner and Hervé Bourlard},
  title = {{Unbiased Semi-Supervised LF-MMI Training Using Dropout}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=6ILzqg6-4bC},
}
Figure