Abstract
In this paper, we presented a novel method for automatic pronunciation quality assessment. Unlike the popular “Goodness of Pronunciation” (GOP) method, this method does not map decoding confidence into a pronunciation quality score but differentiates different pronunciation quality utterances directly. The proposed Two-pass Discriminative Assessment (TPDA) method utilizes a specialized acoustic model trained to distinguish between quality levels at the phoneme level. Experimental results demonstrate that this scoring performance significantly increases compared to the GOP method, particularly at the sentence level.
Introduction
Automatic Pronunciation Quality Assessment (APQA) is a cornerstone of computer-assisted language learning (CALL). For years, the industry standard has been the Goodness of Pronunciation (GOP) method. GOP operates by measuring "decoding confidence"—essentially how closely a student's speech aligns with a native acoustic model. However, this approach is indirect. It requires mapping a continuous confidence score to a discrete human grade (e.g., 0 to 3) using fixed thresholds. Because the acoustic distributions of different quality levels often overlap, setting these thresholds is difficult and prone to error.
The Two-pass Discriminative Assessment (TPDA) Framework
To address the limitations of GOP, we propose the Two-pass Discriminative Assessment (TPDA) method. This approach treats pronunciation quality as a direct classification problem rather than a distance-based metric.
The system operates in two distinct stages:
- Forced Alignment (Pass 1): A standard acoustic model (AM1) is used to perform forced alignment, determining the precise time boundaries for each phone in the student's utterance.
- Discriminative Scoring (Pass 2): A specialized acoustic model (AM2) is employed. In this model, triphones are categorized by their quality level. For instance, a specific triphone is represented by distinct models for Score 1, Score 2, and Score 3.
Discriminative Training and Aggregation
The core innovation lies in how the second acoustic model (AM2) is trained. We utilize Maximum Mutual Information Estimation (MMIE) to ensure the model maximizes the discrimination between identical triphones spoken at different quality levels. This allows the system to learn the specific acoustic signatures of "good" versus "bad" pronunciation.
Furthermore, rather than simply averaging phone-level scores to reach a sentence-level grade, we implement a Maximum Entropy (MaxEnt) model. This data-driven approach weights different triphones based on their reliability as indicators of overall quality, effectively filtering out "noise" from less informative segments of the speech.
Experimental Results
The TPDA method was evaluated using a large-scale dataset of English oral exams from China, comprising approximately 90,000 sentences. The performance was compared against the traditional GOP method using three metrics: Average Scoring Absolute Difference (ASAD), Correlation Coefficient (CC), and Serious Error Rate (SER).
As shown in the figure above, GOP scores (a) are highly scattered, making it difficult to define clear boundaries between human-assigned grades. In contrast, the TPDA method (b) directly classifies the speech into discrete score levels that align closely with human intuition.
Quantitatively, TPDA showed significant improvements:
- ASAD: Reduced from 0.281 (GOP) to 0.219 (TPDA+MaxEnt).
- CC: Increased from 0.797 to 0.843.
- SER: Decreased from 0.102 to 0.085.
Conclusion
The TPDA method represents a significant shift in APQA by treating quality assessment as a discriminative classification task. By training models to recognize the specific features of different quality levels and using MaxEnt for intelligent aggregation, the system achieves a much higher correlation with human graders than traditional confidence-based methods. Future work will focus on addressing data sparsity for rare triphones and refining phone-level labeling.
Summary updated