Full-Sum Decoding for Hybrid Hmm Based Speech Recognition Using LSTM Language Model

Wei Zhou, Wei Zhou 0043, Ralf Schlüter, Hermann NeyICASSP 2020 · DOI

TL;DR

This paper proposes a "full-sum" decoding approach for hybrid HMM-LSTM speech recognition systems, moving beyond the traditional Viterbi approximation. By considering all possible HMM state sequences, they achieve consistent improvements in word error rate across various datasets and models, demonstrating that more accurate probability calculations lead to better decision-making without significant computational overhead.

What Problem Does This Paper Address?

Hybrid HMM-based speech recognition systems combine the strengths of Hidden Markov Models (HMMs) for acoustic modeling and neural networks, particularly Long Short-Term Memory (LSTM) networks, for language modeling. LSTMs can theoretically model unlimited context, which poses a challenge for traditional decoding algorithms. The standard Viterbi approximation in decoding selects the single best HMM state sequence for a given word sequence. However, with LSTMs' ability to capture long-range dependencies, this approximation might discard potentially useful information. The paper questions whether this approximation is truly optimal and explores the benefits of a "full-sum" decoding approach that considers all HMM state sequences to compute the joint probability of a word sequence and acoustic features.

Methods

The core of the paper is the proposal and evaluation of full-sum decoding. Instead of selecting the single best HMM state sequence (Viterbi approximation), full-sum decoding computes the joint probability by summing over all possible HMM state sequences for each word sequence. This is implemented using a modified prefix-tree search algorithm, which is standard for large vocabulary speech recognition. The key modification is replacing the maximization step in the dynamic programming recursion with a summation.

The paper also evaluates different decision rules for speech recognition:

  • Maximum A Posteriori (MAP): A standard decision rule that aims to maximize the posterior probability of a word sequence given the acoustic features.
  • Confusion Network (CN) decoding: An approximation of the general Bayes decision rule, which aims to minimize the expected sentence error rate. The paper uses a pivot-arc-based approach to construct confusion networks.

Datasets

  • Switchboard: A widely used corpus for speech recognition research.
  • LibriSpeech: Another large-scale corpus derived from public domain audiobooks.

The experiments use subsets of these corpora for development (tuning) and testing.

Key Insights and Ideas

  • Full-sum decoding offers tangible benefits: The paper strongly suggests that moving beyond Viterbi approximation and performing a full summation over HMM states leads to improved word error rates. This indicates that the Viterbi approximation, while computationally efficient, can sometimes lead to suboptimal decisions by discarding potentially relevant acoustic paths.
  • Improved probabilities enhance decision-making: The core hypothesis is that more accurate probability calculations (via full-sum) directly translate to better performance in the final decision-making process, even when using standard decision rules like MAP or CN.
  • Full-sum decoding is not computationally prohibitive: While intuitively seeming more complex, the paper argues that when implemented with a prefix-tree search and without recombination (which is already necessary for LSTMs), the computational overhead compared to Viterbi decoding is minimal, primarily involving replacing maximization with summation.
  • Sensitivity to pruning: Full-sum decoding is shown to be more sensitive to aggressive pruning (a technique to reduce search space). However, when pruning is set to a level where WER converges, there's no significant efficiency loss.
  • CN decoding benefits more: The improvements from full-sum decoding are particularly noticeable when used in conjunction with Confusion Network decoding, suggesting that more accurate posteriors are beneficial for this more sophisticated decision rule.

Key Results

  • Consistent improvements: Across various models (trained with Cross-Entropy (CE) and state-level Minimum Bayes Risk (sMBR)) and decision rules (MAP and CN), full-sum decoding consistently achieved lower Word Error Rates (WER) compared to Viterbi decoding on both Switchboard and Librispeech datasets.
  • Specific examples:
    • On Hub5'00, sMBR models with CN decoding saw WER drop from 11.7% (Viterbi) to 11.4% (full-sum).
  • On Librispeech, sMBR models with CN decoding achieved WER of 5.3% (full-sum) compared to 5.4% (Viterbi) on the test-other set.
  • Negligible extra cost: The improvements are achieved without additional training cost, as the full-sum decoding is applied during the recognition phase.

Limitations and Future Work

  • Parameter Tuning: Full-sum decoding requires tuning both acoustic and language model scales (α and β), whereas Viterbi decoding often only needs to tune β. However, the authors note that this effort can be reduced by tuning β first and then linearly scaling both.
  • Sensitivity to Pruning: As mentioned, full-sum decoding is more sensitive to aggressive pruning.
  • Future Directions:
    • Investigate the effect of full-sum decoding in noisy conditions where acoustic models are less confident.
  • Explore the joint effect of full-sum decoding with models trained using full-sum-based criteria like Connectionist Temporal Classification (CTC).

Summary updated

BibTeX
@inproceedings{zhou2020full,
  author = {Wei Zhou and Wei Zhou and Ralf Schlüter and Hermann Ney},
  title = {{Full-Sum Decoding for Hybrid Hmm Based Speech Recognition Using LSTM Language Model}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9053010},
}
Figure