TL;DR
This paper tackles the challenge of poor speech recognition accuracy for non-native speakers by proposing an "online unsupervised acoustic model adaptation" method. By using a "language observer" to softly weigh multiple acoustic models based on predicted language and speaker origin, they achieve a significant 7% improvement in phone-level accuracy for non-native English, French, and Vietnamese speech.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems often struggle with non-native speakers. This is because their pronunciation, intonation, and accent can deviate significantly from the "native" speech patterns the ASR models are trained on. Traditional adaptation methods often require knowing the speaker's native language beforehand (supervised adaptation), which isn't always practical. The core challenge this paper addresses is how to improve ASR accuracy for non-native speakers without prior knowledge of their origin or language, using only the speech signal itself. This is crucial for building more robust and inclusive ASR systems.
Methods
The authors introduce a novel approach called "online unsupervised acoustic model adaptation." Here's a breakdown of their key techniques:
- Multilingual Acoustic-Phonetic Recognizer (MultPR): They start with a baseline ASR system that combines acoustic models for English, French, and Vietnamese. This system is context-independent and uses Hidden Markov Models (HMMs) with Gaussian components.
- Language Observer (LO): This is the core innovation. Instead of making a hard decision about the spoken language (like traditional Language Identification - LID), the LO provides a set of posterior probabilities for each language (e.g., P(English)=0.6, P(French)=0.3, P(Vietnamese)=0.1). This "soft" decision is crucial. The LO is implemented using a "phone recognizer followed by vector space modeling" (PR-VSM) approach.
- Unsupervised Acoustic Model Interpolation: The LO's posterior probabilities are then used as weights to interpolate between the different language acoustic models. This means the system dynamically creates an adapted acoustic model for each utterance, blending the native models based on the LO's assessment of the spoken language and potentially the speaker's native language.
- Two-Pass Decoding: The adaptation happens in two stages. The first pass uses the unadapted MultPR to generate hypotheses and feed them to the LO. The second pass then uses the adapted acoustic model (created by interpolation) for a more accurate recognition.
- INTER-MLLR: They also explore combining their interpolation method with MLLR (Maximum Likelihood Linear Regression), a standard speaker adaptation technique, to see if further gains can be achieved.
Datasets
- MICA Meeting Speech Corpus: This corpus, containing approximately 5 hours of transcribed speech in English (EN), French (FR), Vietnamese (VN), and Khmer (KH), was used to extract test data. It features 14 speakers (1 English, 5 French, 5 Vietnamese, 3 Cambodian) and is noted to be 66% non-native speech.
- WSJ (English), BREF120 (French), VNSpeechCorpus (Vietnamese): These corpora were used for training the monolingual acoustic models that were later combined into the multilingual model. They were also used to train the Vector Space Model (VSM) component of the Language Observer.
The paper focuses on evaluating adaptation for non-native English, French, and Vietnamese speech, extracting utterances longer than 5 seconds.
Key Insights and Ideas
- Soft Decisions are Key: The most significant insight is that using "soft" language posterior probabilities from a language observer (rather than a hard LID decision) is more effective for adaptation. This allows the system to leverage information about multiple potential languages and speaker origins, leading to better adaptation.
- Language Observer Captures More Than Just Spoken Language: The authors hypothesize that their LO can infer not only the spoken language but also hints about the speaker's native language. For example, if P(EN)=0.5, P(FR)=0.4, and P(VN)=0.1, it might suggest English spoken by a French speaker. This nuanced information is valuable for adaptation.
- Online Adaptation is Feasible and Effective: The proposed "online unsupervised" approach, where adaptation happens during decoding without prior knowledge, proves to be a viable strategy for improving non-native ASR.
- Interpolation Outperforms MLLR Alone: The acoustic model interpolation method, guided by the language observer, significantly outperforms the baseline system and even MLLR adaptation alone for non-native speech.
Key Results
- 7% Absolute Improvement: The proposed unsupervised acoustic model interpolation method (INTER) achieved a 7% absolute improvement in phone-level accuracy (PLA) compared to the baseline multilingual system.
- INTER-MLLR Best Performance: Combining interpolation with MLLR (INTER-MLLR) yielded the best average performance, reaching 50.7% PLA, which is even better than using monolingual systems with perfect, pre-detected language identification (48.92% PLA).
- Native vs. Non-Native Performance: While the language observer showed generally poorer accuracy for non-native speech in a direct LID evaluation, the "LID+ORG" metric (which considers the speaker's native language as a correct identification) showed competitive results, suggesting the LO captures useful information.
- Vietnamese Adaptation Sensitivity: The adaptation performance for Vietnamese was more sensitive to the LO's accuracy, indicating that improvements in the LO for specific languages can directly translate to better adaptation results.
Limitations and Future Work
- LO Performance Dependency: The effectiveness of the adaptation is directly tied to the accuracy of the language observer. The paper notes that for Vietnamese, adaptation gains were limited due to the LO's poorer performance on that language.
- Oracle Case for Vietnamese: The paper highlights that in an "oracle case" (perfect LID), Vietnamese adaptation performance was as high as English or French, suggesting that improving the LO's ability to correctly identify Vietnamese and its non-native variants is a key area for future work.
- Future Work: The authors suggest exploring experiments with multilingual acoustic models that share data across languages, which could potentially lead to further improvements. They also implicitly point to the need for more robust language observers, especially for languages with fewer resources or more diverse non-native variations.
Summary updated