Investigation of maximum entropy hybrid language models for open vocabulary German and Polish LVCSR

M. Ali Basha Shaik, Amr El-Desoky Mousa, Ralf Schlüter, Hermann NeyInterspeech 2012

TL;DR

This paper tackles the challenge of recognizing words not present in a speech recognition system's vocabulary (OOVs) for morphologically rich languages like German and Polish. They achieve significant improvements by combining different types of sub-word units (morphemes, syllables, graphones) within hybrid language models and leveraging Maximum Entropy (MaxEnt) models with adaptation techniques.

What Problem Does This Paper Address?

Morphologically rich languages like German and Polish have a vast number of word forms due to inflections, conjugations, and derivations. This leads to a high rate of Out-Of-Vocabulary (OOV) words, meaning words that are not present in the system's lexicon. OOV words are a major hurdle for Large Vocabulary Continuous Speech Recognition (LVCSR) systems, causing recognition errors and impacting overall performance. Traditional LVCSR systems often struggle with these languages because their lexicons cannot possibly encompass all possible word forms. To address this, researchers have explored using sub-word units (like morphemes or syllables) to represent and recognize words, but finding the optimal combination and modeling approach remains a challenge. This paper specifically investigates how to best combine different sub-word units and how to model them effectively using advanced techniques like Maximum Entropy (MaxEnt) language models.

Methods

The authors explore several key methodological advancements:

  • Mixed-unit Sub-words: Instead of relying on a single type of sub-word unit, they propose using a combination of full words, morphemes, syllables, and "graphones" (sub-word units with their context-dependent pronunciations). This allows for a more flexible representation of words.
  • Hybrid Language Models (HLMs): They build language models that integrate these mixed sub-word units. This involves decomposing words into sub-units and then modeling the sequences of these units.
  • Maximum Entropy (MaxEnt) Language Models: They move beyond traditional N-gram models to MaxEnt LMs. MaxEnt models offer greater flexibility by allowing the incorporation of various features, which is crucial for modeling complex sub-word combinations.
  • Adaptation Techniques: To further improve performance, they employ Maximum a-posteriori (MAP) adaptation. This involves adapting pre-trained MaxEnt models to specific in-domain data, either through supervised (using development data) or unsupervised (using transcriptions from a first recognition pass) methods.
  • Interpolation: They combine the strengths of traditional N-gram HLMs and MaxEnt HLMs by interpolating their probabilities, aiming to capture both short-range and long-range dependencies.

Datasets

The experiments were conducted on the following datasets:

  • German: Quaero-2009 development (dev09: 7.5 hours) and evaluation (eval09: 3.8 hours) corpora. These corpora consist of audio from Broadcast News (BN), web, and podcast sources.
  • Polish: Quaero-2010 development (dev10: 3.2 hours) and evaluation (eval10: 3.5 hours) corpora, also comprising BN, web, and podcast audio.
  • Training Corpora: Large corpora were used for language model training: around 188 million running full-words for German (BN) and around 658 million running full-words for Polish (BN, web, podcasts).

Key Insights and Ideas

  • The power of mixed units: Combining different types of sub-word units (morphemes, syllables, graphones) is more effective than using a single type for modeling OOV words in morphologically rich languages.
  • MaxEnt for flexibility: Maximum Entropy models are well-suited for hybrid language modeling because they can effectively incorporate complex features derived from mixed sub-word units, outperforming traditional N-gram models.
  • Adaptation is key for unsupervised gains: While supervised adaptation showed some benefits, unsupervised adaptation (using recognition output) proved particularly effective for improving OOV recognition and reducing overall word error rates, especially for Polish.
  • Graphones offer an edge: Graphones, which combine sub-word units with their pronunciations, showed promise, particularly in the German experiments.

Key Results

  • Polish LVCSR: The best mixed-unit syllabic system (s3) achieved a relative WER reduction of 2.6% on the development set and 3.7% on the evaluation set compared to a 300k full-word baseline. Its performance was comparable to a 500k full-word baseline.
  • German LVCSR: MaxEnt HLMs interpolated with N-gram HLMs showed relative WER reductions of 1.8% compared to a 300k full-word system on the evaluation set. Unsupervised adaptation yielded the best results.
  • Polish LVCSR: The adapted MaxEnt system achieved relative WER reductions of 6.0% compared to a 300k full-word system on the evaluation set.
  • OOV Recognition: The unsupervised adapted system for German recognized around 37% of OOVs, while the Polish mixed-unit system recognized around 36% of OOVs. The best systems outperformed the 500k full-word baselines in terms of overall WER.

Limitations and Future Work

  • Data Sparseness: While mixed-unit models help, the inherent data sparseness in morphologically rich languages remains a challenge.
  • Feature Engineering Complexity: For MaxEnt models, extracting suitable features from mixed sub-word units can be complex due to combinatorial explosion.
  • Resource Constraints: The authors note that resource constraints limited the extent of adaptation experiments (e.g., using 4-gram features for adaptation even when 5-gram features were used for training).
  • Future Work: The paper suggests further exploration of MaxEnt techniques and adaptation for hybrid LMs, potentially with more advanced feature engineering and larger-scale experiments. Investigating more sophisticated methods for sub-word unit selection and combination could also be beneficial.

Summary updated

BibTeX
@inproceedings{shaik2012investigation,
  author = {M. Ali Basha Shaik and Amr El-Desoky Mousa and Ralf Schlüter and Hermann Ney},
  title = {{Investigation of maximum entropy hybrid language models for open vocabulary German and Polish LVCSR}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure