TL;DR
This paper proposes a novel method for improving automatic speech recognition (ASR) systems by merging acoustic models from different languages spoken by the same multilingual individuals. This approach enhances the target language's acoustic model without requiring additional speech data, leading to significant reductions in word error rates.
What Problem Does This Paper Address?
Training robust acoustic models for ASR systems typically requires large amounts of speech data, which can be costly and time-consuming to acquire. While multilingual speech corpora offer a potential solution, existing methods often create suboptimal initial models that need further adaptation with target language data. This paper addresses the challenge of improving a target language's acoustic model by leveraging existing acoustic models from a related source language, specifically when spoken by multilingual individuals. The core idea is that multilingual speakers might share acoustic similarities across languages, which can be exploited for model enhancement.
Methods
The paper introduces a "context-dependent acoustic model merging" technique. Instead of using multilingual models to create an initial target model, they start with an existing target language acoustic model and enhance it using a source language acoustic model. This is achieved by merging the acoustic units (specifically, Gaussian mixtures within HMM states) of the source model into the corresponding states of the target model. The process involves:
- Acoustic Unit Mapping: Determining which acoustic units (phones) in the source language correspond to those in the target language. This is done using the International Phonetic Alphabet (IPA) and perception tests for unique phones.
- Decision Tree Merging: For context-dependent models (triphones), a decision tree is used to group similar triphones. The merging process then involves combining the Gaussians from the source model's states into the corresponding leaf nodes (states) of the target model's decision tree.
- Weighting: A weight is assigned to the source acoustic model (0.1) and the target acoustic model (0.9) during the merging process.
The paper focuses on merging context-dependent acoustic models, specifically triphones, which are more complex than context-independent monophones.
Datasets
- MASS Malay Read Speech Corpus: Used for training the Malay acoustic model. Approximately 120 hours of speech were used.
- MASS Malay Corpus (subset): Used for testing the Malay ASR system.
- Malaysian English Corpus: Used for training and testing the English ASR system. This corpus is described as "non-native" and contains about 9 hours of speech for training and 4 hours for testing.
- Pronunciation Dictionaries:
- MASS pronunciation dictionary (over 60k words) for Malay ASR.
- CMU pronunciation dictionary (over 100k words) for English ASR.
The paper notes that both Malay and English corpora contain speakers of diverse origins (Malay, Chinese, Indian, etc.), reflecting the multicultural nature of Malaysia.
Key Insights and Ideas
- Multilingual Speakers as a Bridge: The fundamental insight is that multilingual speakers' shared acoustic patterns across languages can be leveraged to improve ASR models. This is a departure from traditional cross-lingual transfer methods.
- Enhancing, Not Replacing: The approach focuses on enhancing an existing target model rather than creating a new one from scratch, which is more efficient and potentially more effective.
- Context-Dependent Merging is Key: The paper highlights the complexity and effectiveness of merging context-dependent acoustic models (triphones) compared to simpler monophone models.
- Speaker Origin Matters (Slightly): While the overall improvement is significant, the results show that all speaker origins (Chinese, Malay, Indian) benefit from the merged model, indicating a generalizable improvement.
Key Results
- Word Error Rate (WER) Reduction:
- Malay ASR: Relative WER improvement of 6.8% for monophone merging and up to 9.5% for triphone merging (using 13 MFCC + delta features).
- Malaysian English ASR: Relative WER improvement of 3.6% for monophone merging and up to 4.5% for triphone merging (using 13 MFCC + delta features).
- Model Size Increase: The merging process leads to a significant increase in the number of Gaussians in the acoustic model, sometimes up to four times the original.
- Speaker-Specific Improvements: Table 5 shows that for Malay ASR, all speaker groups (Chinese, Malay, Indian, both male and female) experienced a reduction in WER when using the merged triphone model.
Limitations and Future Work
- Increased Model Size: A significant limitation is the substantial increase in the number of Gaussians in the merged acoustic model, which can impact computational efficiency.
- Future Work: The authors suggest future research directions:
- Reducing Gaussian Count: Developing methods to reduce the number of Gaussians in the merged model without sacrificing performance.
- Speaker Adaptive Acoustic Modeling: Exploring techniques that adapt the merged model more specifically to individual speakers.
Summary updated