TL;DR
This paper tackles the challenge of building large vocabulary speech recognition (LVCSR) systems for agglutinative languages, which create an explosion of word forms. They demonstrate that using statistically derived subword units (morphs) instead of full words significantly improves recognition accuracy and allows for more manageable lexicons.
What Problem Does This Paper Address?
Building effective LVCSR systems, especially for languages like English, relies on vast amounts of transcribed speech and text data to train statistical language models. A key component is the vocabulary size; for English broadcast news, a vocabulary of 50,000 words is common, leading to a low out-of-vocabulary (OOV) rate. However, agglutinative languages, characterized by the concatenation of numerous prefixes, suffixes, and roots, create millions of word forms. This makes it practically impossible to cover all frequent word forms with a manageable lexicon, leading to high OOV rates and poor recognition performance. Traditional rule-based morphological analyzers for splitting words into meaningful parts also suffer from the OOV problem due to handcrafted rules.
Methods
The authors employ an unsupervised, language-independent machine learning method called Morfessor to discover subword units, termed "statistical morphs." Morfessor aims to find units that minimize the coding length of the lexicon and the corpus represented by these units. This approach is advantageous because it requires no handcrafted rules, can process all words (even foreign ones), and tends to produce units that are both frequent and long enough for effective language modeling.
The process involves:
- Extracting Vocabulary: Identifying all unique word forms from a text corpus.
- Morph Segmentation: Using Morfessor to split words into statistical morphs based on learned probabilities.
- Building Morph Lexicon: Creating a lexicon of these discovered morphs.
- Language Model Training: Training n-gram language models using these morphs as the basic units, including special "word break" symbols. Interpolated Kneser-Ney smoothing and entropy-based pruning are used to manage model size and complexity.
Datasets
The paper utilizes text corpora from newspapers, books, and newswire stories for three agglutinative languages:
- Finnish: Approximately 150 million words.
- Estonian: Approximately 55 million words.
- Turkish: Approximately 27 million words.
Speech data for evaluation includes:
- Finnish: Book-reading data from one female speaker (12 hours for training, 27 minutes for testing).
- Estonian: Telephone-recorded sentences from 50 randomly picked speakers.
- Turkish: Newspaper sentence recordings from one female speaker (approximately one hour).
Key Insights and Ideas
- The "Vocabulary Explosion" Problem: Agglutinative languages inherently create a massive number of word forms, making traditional word-based LVCSR lexicons infeasible.
- Morfessor as a Solution: An unsupervised, data-driven approach like Morfessor can effectively discover meaningful subword units (morphs) without language-specific rules.
- Subword Units for Language Modeling: Using these statistical morphs as the basis for n-gram language models significantly reduces lexicon size while maintaining or improving language coverage.
- Language Portability: The Morfessor approach is demonstrated to be effective across three distinct agglutinative languages (Finnish, Estonian, Turkish), highlighting its language independence.
- Performance Gains: Morph-based language models consistently outperform word-based models in LVCSR tasks for these languages.
Key Results
The paper reports significant improvements in Word Error Rate (WER) and Letter Error Rate (LER) for morph-based LVCSR systems compared to word-based reference systems across all three languages:
- Finnish (Speaker-dependent): Morph-based (25k lexicon) achieved 7.0% WER, a substantial improvement over the word-based (400k lexicon) system's 8.5% WER.
- Estonian (Speaker-independent): Morph-based (37k lexicon) achieved 47.6% WER, significantly better than the word-based (60k lexicon) system's 56.3% WER.
- Turkish (Speaker-independent): Morph-based (34k lexicon) achieved 33.9% WER (5-gram), outperforming the word-based system (which showed no improvement beyond 3-grams).
Limitations and Future Work
- Limitations:
- The paper acknowledges that the recognition speed was not a primary concern, but it is a crucial factor for practical applications. Factors like short lexical units, large lexicons, and complex acoustic models can slow down recognition.
- The Turkish language model was not trained using the "growing n-gram" algorithm, and the model was limited to 6-grams, which might affect comparability.
- The study focused on agglutinative languages and general text, and performance might vary for other language types or specialized domains.
- Future Work:
- Further development of grammatical morph-based language models and comparison with the current approach.
- Extending the evaluation to new languages.
- Optimizing recognition speed by potentially using even smaller morph lexicons.
- Investigating the impact of memory constraints on language model size and recognition performance.
Summary updated