Lecture speech recognition by combining word graphs of various acoustic models

Tetsuo Kosaka, Keisuke Goto, Takashi Ito, Masaharu KatoInterspeech 2010

TL;DR

This paper tackles the challenge of improving lecture speech recognition by combining different types of acoustic models (CMHMMs and DMHMMs) using a word graph combination technique. They demonstrate that combining models with distinct error patterns, particularly CMHMM-MMI and DMHMM-MMI, leads to the best performance, suggesting that complementary information is key to robust speech recognition.

What Problem Does This Paper Address?

Large Vocabulary Continuous Speech Recognition (LVCSR) systems, especially for challenging domains like lecture speech, often struggle with performance. While various system combination techniques exist to leverage complementary information from different recognizers, this paper focuses on improving LVCSR by combining acoustic models themselves. The core problem is that individual acoustic models, even when optimized, have limitations and specific error patterns. The authors hypothesize that by intelligently combining models that make different kinds of mistakes, they can achieve better overall recognition accuracy than any single model alone.

Methods

The paper proposes a novel system combination approach that integrates word graphs generated by different acoustic models. The key innovation lies in the selection and combination of acoustic models:

  • Acoustic Models: They utilize two main types of Hidden Markov Models (HMMs):
  • Continuous-Mixture HMMs (CMHMMs): Standard models that represent acoustic features continuously.
  • Discrete-Mixture HMMs (DMHMMs): These models use subvector quantization, making them less susceptible to quantization distortion and potentially offering complementary error patterns.
  • Estimation Criteria: For both CMHMMs and DMHMMs, they explore different parameter estimation techniques: Maximum Likelihood (ML), Maximum A Posteriori (MAP), and Maximum Mutual Information (MMI). Notably, they propose a new MMI estimation technique specifically for DMHMMs.
  • Word Graph Combination: The core combination technique involves:
  • First Pass: Generating word graphs using individual acoustic models (CMHMMs and DMHMMs) with a bigram language model.
  • Combination: Merging these word graphs into a single, combined graph. This is done by essentially taking the union of arcs from individual graphs, preserving timing information.
  • Second Pass (Rescoring): The combined graph is then rescored using a trigram language model. They explore two rescoring strategies: one using a single acoustic model and another using a combination of acoustic models.

Datasets

  • Training and Evaluation: The experiments are conducted on the "Corpus of Spontaneous Japanese" (CSJ).
  • Training: 963 lectures (approximately 203 hours of speech) from 963 speakers were used for acoustic model training.
  • Language Model Training: Text data from 2,668 lectures (6.68M words) was used for training bigram and trigram language models.
  • Test Set: The "testset1" evaluation set, consisting of academic presentations from 10 male speakers (1.7 hours of speech), was used for evaluation.

Key Insights and Ideas

  • Complementarity is Crucial: The central idea is that combining models with different error patterns is more effective than combining similar models. This is empirically supported by the results.
  • DMHMMs Offer Complementary Information: DMHMMs, due to their subvector quantization, exhibit different recognition errors compared to CMHMMs, making them valuable for combination.
  • MMI Estimation for DMHMMs is Beneficial: The proposed MMI estimation for DMHMM parameters leads to improved performance, especially when combined with CMHMMs.
  • Phoneme Mismatch Rate (PMR) as an Indicator: The authors found that the Phoneme Mismatch Rate (PMR) between two models is a good objective indicator of how much improvement can be expected from their combination. A lower PMR suggests more similar error patterns, leading to less benefit from combination.

Key Results

  • Best Performance: The combination of CMHMM-MMI and DMHMM-MMI achieved the best performance, with a Word Error Rate (WER) of 19.88%.
  • Individual Model Performance: Without combination, CMHMM-MMI had a WER of 20.78%, and DMHMM-MMI had a WER of 20.57%.
  • Combination Benefits: Combining CMHMM-MMI and DMHMM-MMI resulted in a significant improvement, reducing the WER by approximately 0.9% compared to the better individual model.
  • Suboptimal Combinations: Combinations of models with similar error patterns, such as DMHMM-MMI and DMHMM-MAP, showed minimal or no improvement.
  • PMR Correlation: Figure 3 clearly illustrates a strong positive correlation between the Phoneme Mismatch Rate (PMR) and the performance improvement rate achieved by combining models.

Limitations and Future Work

  • Limited Number of Systems: The paper explored combinations of two systems. While they mention testing combinations of three or more systems, they concluded that further performance gains were not achieved with the tested models. This suggests that the benefit of adding more systems might diminish or require more sophisticated combination strategies.
  • Future Work: The authors plan to compare their word graph combination approach with other established techniques like ROVER (Recognizer Output Voting Error Reduction) and CNC (Confusion Network Combination). They also aim to investigate more advanced methods for finding effective model combinations.

Summary updated

BibTeX
@inproceedings{kosaka2010lecture,
  author = {Tetsuo Kosaka and Keisuke Goto and Takashi Ito and Masaharu Kato},
  title = {{Lecture speech recognition by combining word graphs of various acoustic models}},
  booktitle = {Interspeech},
  year = {2010},
}

Loading related content...

Figure