Using non-native error patterns to improve pronunciation verification

Joost van Doremalen, Catia Cucchiarini, Helmer StrikInterspeech 2010

TL;DR

This paper introduces a novel pronunciation quality measure, weighted Goodness of Pronunciation (wGOP), that leverages common pronunciation errors made by non-native speakers. By training a logistic regression model on these error patterns, they significantly improve the accuracy of detecting incorrect Dutch vowel pronunciations compared to traditional methods.

What Problem Does This Paper Address?

Learning to pronounce a second language (L2) is a significant challenge for many learners, often leading to difficulties in achieving native-like fluency and intelligibility. While Computer-Assisted Pronunciation Training (CAPT) systems offer a promising solution for providing feedback, a crucial component is the reliable detection of pronunciation errors. Existing methods, like the Goodness of Pronunciation (GOP) measure, estimate the likelihood of a correctly pronounced phoneme but struggle with acoustically similar phonemes, particularly in complex vowel systems like Dutch. This paper aims to bridge this gap by developing a more sensitive pronunciation quality measure that specifically accounts for the types of errors L2 learners tend to make.

Methods

The core of this paper is the development of the weighted Goodness of Pronunciation (wGOP) measure. Unlike the standard GOP, which relies on the maximum likelihood of a phone sequence, wGOP incorporates information about common confusions between phonemes.

Here's a breakdown of their approach:

  • Standard GOP: This measure, often calculated as log{P(O|p)} / max log{P(O|pi)} / dur, estimates the probability of an observed speech segment O given a target phoneme p, normalized by the maximum likelihood over all possible phonemes pi and segment duration. A low GOP score suggests a potential pronunciation error.
  • wGOP: The authors propose a logistic regression model that uses individual vowel likelihood ratios (LLR) as features. These LLRs represent the difference in likelihood between the target vowel and all other potential vowel confusions. The model is trained on manually annotated non-native speech data to learn the weights (β) for each confusion, effectively learning which confusions are most indicative of an error.
  • Model Training: For each target vowel, a separate logistic regression model is trained. The dependent variable is whether the vowel was correctly or incorrectly pronounced. The independent variables are the LLRs of competing vowels. They also experimented with including the standard GOP score as an additional predictor.
  • Evaluation: The performance of GOP, wGOP, and their combination was evaluated using the Equal Error Rate (EER), a standard metric for binary classification tasks.

Datasets

  • JASMIN Speech Corpus: This corpus was used to collect the non-native speech material. It includes recordings from speakers of diverse L1 backgrounds (Turkish, Arabic, Spanish, Chinese, Persian, Hebrew, English) with varying proficiency levels (A1, A2, B1 of the Common European Framework). The experiments utilized read speech material from 45 speakers.
  • Manual Annotations: Orthographic transcriptions were created, and phonetic transcriptions were generated automatically. Crucially, two transcribers manually corrected these phonetic transcriptions to create a gold standard representing how the words were actually realized by the non-native speakers. This manual correction process was essential for identifying and analyzing pronunciation errors.

Key Insights and Ideas

  • Error Patterns Matter: The most significant insight is that simply measuring the likelihood of a correct pronunciation isn't enough. Understanding how non-native speakers err (i.e., which phonemes they confuse) provides crucial information for improving error detection.
  • Dutch Vowel System as a Testbed: The Dutch vowel system, with its numerous acoustically similar vowels in a specific area of the vowel space, presents a challenging scenario. Its complexity makes it an ideal testbed for evaluating the effectiveness of new pronunciation quality measures.
  • Logistic Regression for Error Modeling: Using logistic regression to model the probability of an error based on the likelihood ratios of competing phonemes is a powerful way to leverage confusion patterns. The learned weights (β) directly reflect the diagnostic value of each confusion.
  • Context Dependence and Task Specificity: The authors acknowledge that error patterns can be context-dependent and task-specific. The tuning data used for training the wGOP model needs to be representative of the task (e.g., read speech vs. spontaneous speech) and the L1 backgrounds of the target learners.

Key Results

  • Substantial Performance Improvement: The wGOP measure demonstrated a significant increase in performance compared to the standard GOP measure. For vowels with a high percentage of pronunciation errors and sufficient training data, the improvement was notable.
  • EER Reduction: The EER was reduced by an average of 4.26% for vowels where wGOP outperformed GOP.
  • Benefit for Confused Phonemes: The study found that wGOP particularly benefited the detection of errors involving acoustically close phonemes that are frequently confused. For example, confusions like /y/ with /2:/, /Au/ with /9y/, and /a:/ with /Ei/ showed the most improvement.
  • Combination of GOP and wGOP: Combining the standard GOP score with wGOP provided further, albeit smaller, improvements for some vowels, most notably /e:/ and /I/.
  • Limitations with Low Error Rates: For vowels with very low error rates, the wGOP performance was not as strong, likely due to insufficient data to train reliable regression models.

Limitations and Future Work

  • Generalizability: The authors acknowledge that the generalizability of the trained models to other speakers and tasks is an important concern. While they used diverse L1 backgrounds, further investigation into L1-specific tuning is suggested.
  • Task Appropriateness: The tuning data must be appropriate for the specific task (e.g., read speech vs. spontaneous speech) as error patterns can differ.
  • Equal Treatment of Errors: The current approach treats all pronunciation errors equally. Future work could explore differentiating the severity of errors (e.g., /2:/ as /y/ vs. /2:/ as /@/) and adjusting thresholds accordingly.
  • Context Dependence: Handling context-dependent pronunciation variations for some vowel sounds is a planned area of future research.
  • Segmentation Accuracy: The accuracy of the initial phonetic segmentation heavily influences confidence scoring, and improvements in this area are also considered.
  • Speaker Adaptation: Speaker adaptation of HMM models is another avenue for future improvement.
  • Generalization to Consonants: The authors plan to investigate how their method generalizes to consonant sounds.

Summary updated

BibTeX
@inproceedings{doremalen2010using,
  author = {Joost van Doremalen and Catia Cucchiarini and Helmer Strik},
  title = {{Using non-native error patterns to improve pronunciation verification}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure