Recognize Mispronunciations to Improve Non-Native Acoustic Modeling Through a Phone Decoder Built from One Edit Distance Finite State Automaton

Wei Chu, Yang Liu, Jianwei Zhou2020 · DOI

TL;DR

This paper introduces a novel method to improve acoustic models for non-native English speakers by automatically detecting and correcting mispronunciations in training data. By using a constrained phone decoder and Goodness-of-Pronunciation (GOP) scores, they refine phone alignments, leading to a significant reduction in Word Error Rate (WER) for non-native speech recognition.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems often struggle with non-native accents, particularly those of English as a Second Language (ESL) learners. This difficulty stems not only from pronunciation variations (accent) but also from actual mispronunciations of phonemes. Traditional acoustic modeling relies on accurate transcriptions and phone alignments. When training data contains mispronunciations, the acoustic model learns incorrect associations between speech features and phonemes, degrading recognition performance. Existing methods for detecting mispronunciations often require expert linguistic knowledge or are prone to errors, especially when dealing with ASR-generated transcriptions. This paper aims to address this by developing a data-driven approach to automatically identify and correct mispronunciations in training data, thereby improving the robustness of acoustic models for non-native speakers.

Methods

The core of the proposed method involves an iterative process of mispronunciation detection and recognition.

  1. Initial Phone Alignment: A baseline phone sequence is generated from the word-level transcription and a canonical pronunciation dictionary.
  2. Mispronunciation Detection (GOP-based): Goodness-of-Pronunciation (GOP) scores are calculated using a GMM-HMM system trained on native speech. Phones with low GOP scores (indicating potential mispronunciation) are flagged.
  3. Constrained Mispronunciation Recognition: For flagged phones, a specialized Finite State Automaton (FSA) is used. This FSA is designed to search for phone sequences that are at most one edit distance (substitution, insertion, or deletion) away from the original phone. Crucially, this FSA has a "no loop-back" design, which limits the search space and complexity. This constrained decoder helps to recognize the most likely intended phone sequence.
  4. Alignment Update: If a mispronunciation is recognized and corrected, the phone alignments and GOP scores are updated.
  5. Iteration: Steps 2-4 are repeated until no more mispronunciations are detected based on a sequence GOP (S-GOP) score criterion.
  6. Acoustic Model Training: The refined, corrected phone alignments are then used to train a non-native acoustic model (TDNN-based).

The process is performed twice: first using a native GMM for GOP scoring and alignment, followed by a speaker-adaptive training of a non-native GMM, and then a second round of GOP-based alignment using the non-native GMM. This two-round approach is found to yield better alignments.

Datasets

  • Training Data: A 300-hour non-native spontaneous speech dataset from 2.8k Chinese ESL learners.
  • Auxiliary Data: A 644-hour read speech dataset from 1.3k native speakers, used solely for training a native GMM for GOP scoring.
  • Test Data: A 3k utterance test set from 56 non-native speakers.

Key Insights and Ideas

  • Data-driven Mispronunciation Correction: The paper demonstrates that mispronunciations in training data can be automatically detected and corrected without requiring explicit linguistic rules or expert annotations.
  • Constrained Decoder for Efficiency and Accuracy: The novel "no loop-back" FSA significantly reduces the search space for mispronunciation recognition, making the process computationally feasible while maintaining accuracy. This is a key departure from unconstrained decoders.
  • GOP as a Reliable Indicator: Goodness-of-Pronunciation scores, derived from native speech models, prove to be effective indicators for flagging potentially mispronounced phones in non-native speech.
  • Iterative Refinement: The iterative nature of the detection and correction process allows for progressive improvement of the phone alignments, leading to a more robust acoustic model.
  • Value of Native Speech Models: Leveraging acoustic models trained on native speech for GOP scoring is crucial for accurately identifying deviations in non-native speech.

Key Results

  • The proposed method achieved a 2% relative reduction in Word Error Rate (WER) compared to a well-optimized baseline TDNN-HMM system trained on the same non-native data.
  • When combined with the baseline system using lattice interpolation, the overall WER was reduced by 6%.
  • The error breakdown showed a significant 8% drop in insertion errors and a 6% drop in substitution errors.
  • The training process with the proposed method exhibited a faster loss drop and a lower final loss on both training and validation sets compared to the baseline.
  • The detected mispronunciation rate in the training data was 3.8%, with substitution errors being the most common (75.4%), followed by insertion (23.6%) and deletion (1.0%).

Limitations and Future Work

  • Threshold Sensitivity: The mispronunciation detection relies on a threshold ($\alpha$) for the S-GOP score, which was set empirically. A higher threshold was used to focus on more significant mispronunciations, potentially missing some actual errors.
  • Assumption of Neighboring Phone Accuracy: The method assumes that phones adjacent to a mispronounced phone are not themselves mispronounced. While the iterative process can loosen this assumption, it remains a potential limitation.
  • GMM-based GOP: The paper uses GMM-based GOP scores, acknowledging that DNN-based or Forced-GOP algorithms might yield better scores. However, GMM-based alignments are still considered sufficient as intermediate indicators.
  • Future Work: The authors suggest that the identified mispronunciation patterns could be valuable for speech assessment tools. Further research could explore more advanced decoding techniques or integrate other acoustic features for improved mispronunciation detection and recognition. The paper also notes that combining their system with a baseline using lattice interpolation yielded better results, suggesting that their method captures different error patterns than the baseline.

Summary updated

BibTeX
@misc{chu2020recognize,
  author = {Wei Chu and Yang Liu and Jianwei Zhou},
  title = {{Recognize Mispronunciations to Improve Non-Native Acoustic Modeling Through a Phone Decoder Built from One Edit Distance Finite State Automaton}},
  year = {2020},
  doi = {10.21437/interspeech.2020-3109},
}
Figure