Analysing Recognition Errors in Unlimited-Vocabulary Speech Recognition

Teemu Hirsimäki, Mikko KurimoNAACL 2009

TL;DR

This paper delves into the error patterns of a Finnish unlimited-vocabulary speech recognition system, revealing how different training techniques like speaker adaptive training (SAT) and discriminative training (MPFE) impact error types. The findings highlight that while SAT primarily benefits acoustic model-driven errors, MPFE helps with language model errors, and that a significant portion of errors still occur on common words, suggesting avenues for future improvement.

What Problem Does This Paper Address?

Building robust speech recognition systems, especially those handling a vast vocabulary, is a complex endeavor. Understanding why these systems make mistakes is crucial for targeted improvements. Previous work has analyzed errors, but this paper focuses on a morph-based Finnish system with an effectively unlimited vocabulary, examining how specific training strategies influence different categories of recognition errors. The challenge is to dissect the contributions of acoustic and language models to errors and identify the most impactful areas for future system development.

Methods

The authors employ Error Region Analysis (ERA) to categorize errors based on whether the acoustic model (AM), language model (LM), or both are responsible for a misrecognition. They then analyze these error regions in conjunction with manual error classification and word frequency analysis.

Key training techniques investigated include:

  • Maximum Likelihood (ML) training: The baseline acoustic model.
  • Speaker Adaptive Training (SAT): Uses constrained maximum likelihood linear regression (CMLLR) to adapt models to speakers, improving robustness to speaker variability.
  • Discriminative Training with Minimum Phone Frame Error (MPFE): A criterion that directly optimizes for minimizing errors at the phone level, aiming to improve the overall recognition accuracy.

The language model is a high-order Kneser-Ney smoothed n-gram model built using morphs derived from the Finnish Kielipankki corpus.

Datasets

  • Language Model Training Data: 150 million words from the Finnish Kielipankki corpus. Words were split into morphs using the Morfessor algorithm.
  • Acoustic Phoneme Models Training Data: Finnish SpeechDat telephone speech database.
  • Training: 39 hours from 3838 speakers.
  • Development: 46 minutes from 79 speakers.
  • Evaluation: A similar set of 46 minutes from 79 speakers.

Key Insights and Ideas

  • SAT and MPFE target different error types: SAT is particularly effective at reducing errors where the acoustic model is the primary culprit (HYP-AM regions). MPFE, on the other hand, helps reduce errors influenced by the language model (HYP-LM and HYP-BOTH regions).
  • Common words are still a major error source: Despite advancements, a substantial number of letter errors occur on the 1000 most frequent words in the vocabulary. This suggests that improving recognition for high-frequency words remains a critical area.
  • Foreign words and acronyms are challenging: The lack of specific modeling for foreign pronunciations and acronyms leads to a significant number of errors in these categories. This is a clear area for future development.
  • Manual error classification is insightful: Categorizing errors manually (e.g., "Foreign," "Inflect," "Poor," "Noise") provides a deeper understanding of the underlying causes, beyond just AM vs. LM blame.

Key Results

  • Overall Error Reduction: The ML+SAT+MPFE system achieved a Letter Error Rate (LER) of 4.8%, down from 6.8% for the ML system. SAT alone reduced LER to 5.6%.
  • SAT's Impact: SAT significantly reduced errors in the HYP-AM class, as expected, by adapting to speaker variations.
  • MPFE's Impact: MPFE further reduced errors in HYP-BOTH and HYP-LM classes, complementing SAT's effects.
  • Manual Error Breakdown (ML+SAT+MPFE system):
    • "Foreign" names accounted for about 8% of letter errors.
  • "Inflect" errors (slight variations in word endings) were also notable.
  • "Poor" pronunciation and "Noise" were identified as difficult-to-improve categories.
  • Frequency Analysis: While error rates increase for infrequent and out-of-vocabulary words, the absolute number of letter errors is highest for the 1000 most frequent words.

Limitations and Future Work

  • Limitations:
    • The morph-based recognizer currently lacks specific modeling for foreign pronunciations and acronyms, leading to predictable errors.
  • Distinguishing between the intended word and the actually spoken word in cases of "Poor" pronunciation or repairs is challenging.
  • Compound word errors can be tricky, sometimes due to ambiguity in word boundaries or even errors in the language model training data itself.
  • Future Work:
    • Improving pronunciation modeling for foreign words and acronyms is a key recommendation.
  • Developing methods to detect when acoustic evidence is unreliable and to place more trust in the language model could help address errors on common words.
  • Further investigation into compound word errors and their handling in the language model is suggested.

Summary updated

BibTeX
@inproceedings{hirsimaki2009analysing,
  author = {Teemu Hirsimäki and Mikko Kurimo},
  title = {{Analysing Recognition Errors in Unlimited-Vocabulary Speech Recognition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure