TL;DR
This paper explores using subword units (morphs) instead of whole words for language modeling in speech recognition, particularly for morphologically rich languages. They demonstrate that morph-based models significantly improve recognition accuracy by better handling out-of-vocabulary words and reducing data sparsity, outperforming traditional word-based models across most tested languages.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems rely heavily on language models (LMs) to predict the most likely sequence of words. However, traditional LMs are trained on word sequences, which struggle with "morphologically rich" languages. These languages, like Finnish, Turkish, and Arabic, have a vast number of word forms generated through processes like inflection, derivation, and compounding. This leads to a high rate of "out-of-vocabulary" (OOV) words – words encountered during testing that were not seen during training. This OOV problem severely degrades ASR performance. The paper aims to address this by investigating whether modeling language at the subword (morph) level can overcome the limitations of word-based LMs in these complex languages.
Methods
The core of the paper's methodology involves using the Morfessor Baseline algorithm, an unsupervised, data-driven approach to segment words into morpheme-like units. This algorithm aims to find an optimal balance between the size of the morph lexicon and how well the corpus can be represented by these morphs.
- Morph Segmentation: Morfessor identifies frequently occurring substrings as potential morphs and represents words as sequences of these morphs.
- Morph-based Language Models: Instead of training n-gram LMs on word sequences, they train them on sequences of these identified morphs. This allows the model to construct and recognize previously unseen word forms by combining familiar morphs.
- Comparison: The performance of these morph-based LMs is directly compared against standard word-based n-gram LMs.
- Analysis: The evaluation is conducted by partitioning the test sets into "in-vocabulary" (IV) and "out-of-vocabulary" (OOV) words to understand how each model type performs on known versus unknown word forms.
Datasets
The experiments were conducted across four morphologically rich languages:
- Finnish: Multiple datasets (Fin1, Fin2, Fin3, Fin4) were used, including news broadcasts and audiobooks, with both speaker-dependent and speaker-independent acoustic models.
- Estonian: Data consisted of newspaper sentences, names, and digits, with speaker-independent acoustic models.
- Turkish: Speech data was from newspaper text read by a single speaker, using speaker-independent acoustic models.
- Egyptian Colloquial Arabic (ECA): This dataset comprised transcribed spontaneous telephone conversations, characterized by disfluencies, and used speaker-independent acoustic models.
LM training corpora varied in size, with Finnish, Estonian, and Turkish using substantial amounts of news and book texts, while Arabic used a relatively small corpus of transcribed spontaneous speech.
Key Insights and Ideas
- Morphs Generalize Better: The fundamental idea is that by breaking words into smaller, reusable units (morphs), the system can construct and understand novel word forms it has never seen before, a capability word-based models lack.
- OOV is the Key Battleground: The paper strongly suggests that the primary advantage of morph-based models lies in their ability to handle OOV words, rather than necessarily improving accuracy on words already in the vocabulary.
- Language-Specific Benefits: While morph models generally outperform word models, the degree of benefit varies across languages. Finnish and Estonian showed the most significant gains, likely due to their highly agglutinative nature.
- Arabic's Unique Challenge: Arabic, despite being morphologically rich, showed less benefit from morph models, potentially due to its templatic morphology and the limited size/style of the training data.
- "New Words" are Tough: Even morph models struggle significantly with words that are entirely new, not just to the word lexicon but also not constructible from learned morphs in observed ways.
Key Results
- Overall Accuracy: Morph-based LMs achieved higher word accuracies than word-based LMs in most configurations, with the exception of Egyptian Colloquial Arabic (ECA), where the word model performed better.
- In-Vocabulary (IV) Words: Morph models performed comparably to word models on IV words, meaning they didn't sacrifice accuracy on known words to gain OOV coverage.
- Out-of-Vocabulary (OOV) Words: Morph models showed significantly better accuracy on OOV words compared to word models, which often resulted in negative accuracies (more insertions than correct recognitions).
- New Words: Both model types performed poorly on entirely new words, often resulting in negative accuracies, indicating the difficulty of recognizing words not seen or constructible from training data.
Limitations and Future Work
- Arabic Performance: The paper notes that the morph model's performance for Arabic was not as beneficial as expected, possibly due to the language's templatic morphology and the limited size and spontaneous nature of the training data.
- "New Words" Challenge: The authors acknowledge that recognizing entirely unseen words, which require novel combinations of morphs, remains a significant challenge for both model types.
- Factored Language Models (FLMs): For Arabic, they mention that FLMs, which incorporate morphological lexicons and handle templatic morphology, barely outperformed standard word models, suggesting potential but also complexity.
- Future Directions: The paper implies that for highly complex languages or to further push performance, more advanced techniques like Factored Language Models or refined subword units might be necessary complements to standard n-grams. The authors also suggest that simply expanding word vocabularies to cover more words is not always feasible or efficient.
Summary updated