Error Pattern Detection Integrating Generative and Discriminative Learning for Computer-Aided Pronunciation Training

Yow-Bang Wang, Lin-Shan LeeInterspeech 2012 · DOI

Introduction: The Challenge of Error Pattern Detection

In the field of Computer-Aided Pronunciation Training (CAPT), providing effective feedback requires more than just identifying a general error; the system must pinpoint how a learner's pronunciation deviates from the target. These specific mistakes are known as Error Patterns (EPs). For instance, a native English speaker learning Mandarin might consistently substitute a specific Mandarin phoneme with a similar-sounding English one.

The technical difficulty lies in the fact that EPs are often acoustically very similar to the correct (canonical) pronunciation. Traditional generative models, such as Hidden Markov Models (HMMs), excel at modeling general speech characteristics but often lack the discriminative power to distinguish between a correct phoneme and a subtle error. This frequently results in high False Acceptance Rates (FAR), where the system incorrectly validates mispronounced speech, undermining the learning process.

Proposed Methodology: A Hybrid Two-Pass Framework

To address the limitations of purely generative models, this paper proposes a hybrid framework that integrates the strengths of HMM/GMMs with the sharp classification boundaries of Support Vector Machines (SVMs). The system utilizes a two-pass Viterbi decoding architecture to refine error detection.

System architecture for two-pass Viterbi decoding integrating generative and discriminative models.

As illustrated in the architecture above, the process consists of:

  1. First Pass: A standard Viterbi decoding (forced alignment) determines the precise time boundaries of each phoneme in the learner's utterance.
  2. Second Pass: A sophisticated decoding pass where the acoustic score is calculated as a weighted sum of a Generative Score (log-likelihood from an HMM/GMM) and a Discriminative Score (log-posterior probability from an SVM with an RBF kernel).

The researchers evaluated three distinct SVM integration schemes:

  • $(N_p+1)$-ary SVM: A single multi-class classifier distinguishing between correct pronunciation and all possible EPs for a specific phoneme.
  • Binary SVM: A simplified classifier trained strictly to distinguish "correct" from "incorrect."
  • Cascaded SVMs: A two-stage approach where a binary SVM first identifies if a segment is correct or incorrect; if flagged as incorrect, a second multi-class SVM identifies the specific EP.

Experimental Setup

The framework was evaluated using a comprehensive Learner Corpus containing recordings from 278 Mandarin learners across 36 countries. The data was partitioned into training/adaptation (8.4 hours), development (2.2 hours), and testing (2.0 hours) sets. Native base models were initialized using the ASTMIC corpus (24.6 hours of native Mandarin), while English phoneme models (to define cross-linguistic EPs) were trained on the TIMIT corpus. Expert language teachers manually annotated 152 distinct Error Patterns across 39 Mandarin phonemes to provide ground truth.

Results and Performance Analysis

The experimental results demonstrate that the Cascaded SVM framework significantly outperforms the baseline generative model across several key metrics.

Performance comparison of different SVM integration schemes on error detection metrics.

As shown in the table above, the Cascaded SVM approach achieved the following:

  • Average False Acceptance Rate (AFAR): Reduced from 60.70% (baseline) to 41.52%, representing a substantial improvement in the system's ability to catch learner mistakes.
  • Average Error Rate (AER): Improved from 36.37% to 33.43%.
  • Diagnosis Accuracy: The number of correctly diagnosed EPs rose from 1126 to 1330.

However, this increased sensitivity introduced a trade-off: the Average False Rejection Rate (AFRR) rose from 12.03% to 25.34%. This indicates that while the system is much better at catching errors, it has become significantly stricter, occasionally flagging correct pronunciations as incorrect.

Conclusion and Future Directions

Integrating discriminative SVM scores into a generative HMM framework effectively addresses the "forgiving" nature of standard HMMs in CAPT systems. The Cascaded SVM approach proved particularly effective by decoupling the detection of an error from its specific diagnosis, thereby mitigating issues related to data imbalance between correct and erroneous samples.

Future work will focus on balancing the system's strictness to avoid discouraging learners due to false rejections. Potential avenues include exploring discriminative training for the HMM/GMMs themselves and incorporating "Goodness of Pronunciation" (GOP) scores into the integrated framework to further refine detection accuracy.

Summary updated

BibTeX
@inproceedings{wang2012error,
  author = {Yow-Bang Wang and Lin-Shan Lee},
  title = {{Error Pattern Detection Integrating Generative and Discriminative Learning for Computer-Aided Pronunciation Training}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-249},
}
Figure