An Empirical Study on Multiple LVCSR Model Combination by Machine Learning

Takehito Utsuro, Yasuhiro Kodama, Tomohiro Watanabe, Hiromitsu Nishizaki, Seiichi NakagawaNAACL 2004

TL;DR

This paper explores a machine learning approach to combine outputs from multiple Large Vocabulary Continuous Speech Recognition (LVCSR) models. By using Support Vector Machines (SVMs), they achieve better word error reduction than traditional voting methods like ROVER, especially when individual models are unreliable.

What Problem Does This Paper Address?

Speech recognition systems, while impressive, are not perfect and often make errors. When using multiple speech recognition models, simply combining their outputs through methods like voting (e.g., ROVER) can be effective. However, these voting schemes can be negatively impacted if many of the participating models are not very accurate. The challenge is to find a more robust way to leverage the strengths of multiple LVCSR models, particularly when some of them might be less reliable, to improve overall recognition accuracy.

Methods

The core of this paper is the application of Support Vector Machine (SVM) learning to combine the outputs of multiple LVCSR models. Instead of a simple majority vote, SVMs are trained to intelligently select the most confident word among several hypotheses. The key innovation lies in the features used for SVM training:

  • Model IDs: Which specific LVCSR model produced the hypothesized word.
  • Part-of-speech: The grammatical role of the word.
  • Number of syllables: The phonetic structure of the word.

These features allow the SVM to learn which models are more reliable for certain types of words and to combine their outputs in a more optimal way than traditional voting. The SVM essentially learns to discriminate between correct and incorrect word hypotheses, using the distance from the separating hyperplane as a measure of confidence.

Datasets

The paper uses two primary evaluation datasets:

  • Newspaper sentence utterances: 100 utterances from 10 male speakers, totaling 1,565 words. This data is considered relatively easier for speech recognizers.
  • Broadcast news speech: 175 Japanese broadcast news sentences from 14 male speakers (announcers and reporters), totaling 6,813 words. This data is considered harder.

For training and testing, each of these datasets was divided into two halves, ensuring that speakers were not shared between the training and testing sets.

Key Insights and Ideas

  • SVMs offer a more nuanced combination: Unlike voting methods that treat all models equally or with simple weights, SVMs can learn to identify and down-weight unreliable models based on features like model ID and word characteristics.
  • Model selection matters: Choosing models that collectively cover a wide range of correctly recognized words (maximizing recall of the union) is more effective than simply picking the best-performing models in descending order of their individual word correct rates. This is because SVMs can handle the noise from less reliable models in the combined set.
  • Confidence is key: The SVM approach leverages confidence measures derived from the model combination process, which can be higher than those from individual models.

Key Results

The proposed SVM-based model combination technique demonstrated significant improvements:

  • Newspaper utterances: Achieved a relative word error reduction of up to 39% compared to the best single model and 23% compared to ROVER.
  • Broadcast news speech: Achieved a relative word error reduction of up to 13% compared to the best single model and 8% compared to ROVER.
  • Remarkable improvements were particularly noted in word accuracy rates, especially when a confidence threshold was applied to discard less certain hypotheses, effectively reducing insertion errors.

Limitations and Future Work

  • Model selection strategy: While maximizing recall of the union was shown to be better than descending order of word correct rates, the paper suggests further empirical investigation into optimal model selection strategies.
  • Feature contribution: The paper notes that the contribution of part-of-speech and syllable count features to SVM learning was slight, suggesting that model IDs and acoustic/language scores might be more dominant. Further analysis into feature importance could be beneficial.
  • Generalizability: The study focuses on Japanese LVCSR. While the principles are likely generalizable, further validation on other languages and acoustic conditions would be valuable.

Summary updated

BibTeX
@inproceedings{utsuro2004empirical,
  author = {Takehito Utsuro and Yasuhiro Kodama and Tomohiro Watanabe and Hiromitsu Nishizaki and Seiichi Nakagawa},
  title = {{An Empirical Study on Multiple LVCSR Model Combination by Machine Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}
Figure