A Novel Discriminative Method for Pronunciation Quality Assessment

Junbo Zhang, Fuping Pan, Bin Dong, Qingwei Zhao, Yonghong Yan

Abstract

In this paper, we presented a novel method for automatic pronunciation quality assessment. Unlike the popular “Goodness of Pronunciation” (GOP) method, this method does not map decoding confidence into a pronunciation quality score but differentiates different pronunciation quality utterances directly. The proposed Two-pass Discriminative Assessment (TPDA) method utilizes a specialized acoustic model trained to distinguish between quality levels at the phoneme level. Experimental results demonstrate that this scoring performance significantly increases compared to the GOP method, particularly at the sentence level.

Introduction

Automatic Pronunciation Quality Assessment (APQA) is a cornerstone of computer-assisted language learning (CALL). For years, the industry standard has been the Goodness of Pronunciation (GOP) method. GOP operates by measuring "decoding confidence"—essentially how closely a student's speech aligns with a native acoustic model. However, this approach is indirect. It requires mapping a continuous confidence score to a discrete human grade (e.g., 0 to 3) using fixed thresholds. Because the acoustic distributions of different quality levels often overlap, setting these thresholds is difficult and prone to error.

The Two-pass Discriminative Assessment (TPDA) Framework

To address the limitations of GOP, we propose the Two-pass Discriminative Assessment (TPDA) method. This approach treats pronunciation quality as a direct classification problem rather than a distance-based metric.

The system architecture of the TPDA method, illustrating the two-pass decoding process and final MaxEnt scoring.

The system operates in two distinct stages:

  1. Forced Alignment (Pass 1): A standard acoustic model (AM1) is used to perform forced alignment, determining the precise time boundaries for each phone in the student's utterance.
  2. Discriminative Scoring (Pass 2): A specialized acoustic model (AM2) is employed. In this model, triphones are categorized by their quality level. For instance, a specific triphone is represented by distinct models for Score 1, Score 2, and Score 3.

Discriminative Training and Aggregation

The core innovation lies in how the second acoustic model (AM2) is trained. We utilize Maximum Mutual Information Estimation (MMIE) to ensure the model maximizes the discrimination between identical triphones spoken at different quality levels. This allows the system to learn the specific acoustic signatures of "good" versus "bad" pronunciation.

Furthermore, rather than simply averaging phone-level scores to reach a sentence-level grade, we implement a Maximum Entropy (MaxEnt) model. This data-driven approach weights different triphones based on their reliability as indicators of overall quality, effectively filtering out "noise" from less informative segments of the speech.

Experimental Results

The TPDA method was evaluated using a large-scale dataset of English oral exams from China, comprising approximately 90,000 sentences. The performance was compared against the traditional GOP method using three metrics: Average Scoring Absolute Difference (ASAD), Correlation Coefficient (CC), and Serious Error Rate (SER).

Comparison of (a) GOP scores and (b) TPDA scores. The TPDA method provides much clearer separation aligned with human grades.

As shown in the figure above, GOP scores (a) are highly scattered, making it difficult to define clear boundaries between human-assigned grades. In contrast, the TPDA method (b) directly classifies the speech into discrete score levels that align closely with human intuition.

Quantitatively, TPDA showed significant improvements:

  • ASAD: Reduced from 0.281 (GOP) to 0.219 (TPDA+MaxEnt).
  • CC: Increased from 0.797 to 0.843.
  • SER: Decreased from 0.102 to 0.085.

Conclusion

The TPDA method represents a significant shift in APQA by treating quality assessment as a discriminative classification task. By training models to recognize the specific features of different quality levels and using MaxEnt for intelligent aggregation, the system achieves a much higher correlation with human graders than traditional confidence-based methods. Future work will focus on addressing data sparsity for rare triphones and refining phone-level labeling.

Summary updated

Abstract

In this paper, we presented a novel method for automatic pronunciation quality assessment. Unlike the popular “Goodness of Pronunciation” (GOP) method, this method does not map the decoding confidence into pronunciation quality score, but differentiates the different pronunciation quality utterances directly. In this method, the student's utterance need to be decoded for two times. The first-time decoding was for getting the time points of each phone of the utterance by a forced alignment using a conventional trained acoustic model (AM). The second-time decoding was for differentiating the pronunciation quality for each triphone using a specially trained AM, where the triphones in different pronunciation qualities were trained as different units, and the model was trained in discriminative method to ensure the model has the best discrimination among the triphones whose names were same but pronunciation quality scores were different. The decoding network in the second-time decoding included different pronunciation quality triphones, so the phone-level scores can be obtained from the decoding result directly. The phone-level scores were combined into the sentence-level scores using maximum entropy criterion. The experimental results shows that the scoring performance was increased significantly compared to the GOP method, especially in sentence-level.
BibTeX
@inproceedings{zhang2013novel,
  author = {Junbo Zhang and Fuping Pan and Bin Dong and Qingwei Zhao and Yonghong Yan},
  title = {{A Novel Discriminative Method for Pronunciation Quality Assessment}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2013},
  doi = {10.1109/icassp.2013.6639268},
}
Figure