Improvement in automatic pronunciation scoring using additional basic scores and learning to rank

Liang-Yu Chen, Jyh-Shing Roger JangInterspeech 2012

TL;DR

This paper improves automatic pronunciation scoring by incorporating novel vowel- and consonant-specific word-level scores and applying learning-to-rank (LTR) algorithms. The approach significantly outperforms existing methods, demonstrating the value of finer-grained phonetic analysis and advanced machine learning for evaluating spoken language proficiency.

What Problem Does This Paper Address?

Automatic Pronunciation Scoring (APS) systems are crucial for Computer-Assisted Pronunciation Training (CAPT) by providing automated feedback on learners' spoken English. Existing APS systems often rely on a single type of "basic score" (e.g., HMM-based log-likelihood) or simple averaging of phone-level scores to derive a word-level score. This can overlook the distinct contributions of vowels and consonants to overall pronunciation quality. Furthermore, combining multiple basic scores effectively has been a challenge, with methods like neural networks being computationally expensive. This paper aims to address these limitations by exploring more informative word-level features and leveraging learning-to-rank techniques for more robust and efficient score combination.

Methods

The core of this work lies in two main areas:

  • Enhanced Word-Level Scores: The authors extend their previous work by converting phone-level basic scores into word-level scores using three methods:
  • Average-based: The standard method, averaging all phone scores within a word.
  • Vowel-based: Averaging only the phone scores of vowel segments within a word.
  • Consonant-based: Averaging only the phone scores of consonant segments within a word. This creates 18 distinct word-level scores (6 basic scores x 3 conversion methods).

  • Learning to Rank (LTR): Instead of directly combining scores into a single output, LTR algorithms are used to learn a ranking function. This approach treats pronunciation scoring as an information retrieval problem, where the goal is to order utterances by their quality. The paper compares two LTR approaches:

  • Pairwise (RankSVM): Learns by comparing pairs of utterances.
  • Listwise (ListNet): Learns by considering the entire list of utterances at once.

The system also includes an LTR score quantization stage to convert continuous LTR scores into discrete ranks.

Datasets

  • Native Corpus: The Wall Street Journal (WSJ) corpus was used to train Hidden Markov Models (HMMs) for speech recognition and to obtain necessary phonetic statistics.
  • Non-Native Corpus: The MIR-SD dataset was used for training and evaluation. This dataset consists of 1100 utterances from 22 Taiwanese speakers, each speaking over 200 multi-syllabic English words. Human rankings on a scale of 1 (unintelligible) to 5 (native-like) were collected for this corpus, with reported word-based inter-rater correlation of 0.58 and speaker-based correlation of 0.78.

Key Insights and Ideas

  • Vowel and Consonant Importance: The paper highlights that treating vowels and consonants separately when deriving word-level scores can capture nuances missed by simple averaging, leading to improved pronunciation assessment.
  • Learning to Rank Efficacy: LTR methods, particularly ListNet, demonstrate superior performance compared to traditional score combination techniques like Gaussian classifiers or linear combinations.
  • Computational Efficiency: The listwise LTR approach (ListNet) offers a significant advantage in terms of training time compared to the pairwise RankSVM, making it more practical for real-world applications.
  • Feature Combination Benefits: Combining all three types of word-level scores (average, vowel, consonant) generally leads to better performance, especially when used with LTR methods.

Key Results

  • The proposed LTR-based framework with enhanced word-level scores significantly outperforms most existing methods.
  • Specifically, the ListNet approach, when using all 18 word-level scores, achieved the best performance across various evaluation metrics (raw correlation, rank correlation, rank recognition rate, etc.).
  • The LTR methods (RankSVM and ListNet) performed better than traditional score combination methods like GCE, LCA, and NN.
  • The best basic scoring method, HMM-based log-posterior probability scores (hmmPost), was surpassed by the LTR-based approaches.
  • Training time for RankSVM was considerable (4-6 hours), while ListNet was much faster (a few seconds).

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the MIR-SD dataset, while novel, is relatively small. They plan to test their methods on a larger dataset currently under construction.
  • LTR Algorithm Exploration: The paper focused on RankSVM and ListNet. Future work will explore other LTR algorithms, such as RBF-ListNet and RankNet.
  • Generalizability: While the results are promising, further validation on diverse non-native speaker populations and different languages would be beneficial.

Summary updated

BibTeX
@inproceedings{chen2012improvement,
  author = {Liang-Yu Chen and Jyh-Shing Roger Jang},
  title = {{Improvement in automatic pronunciation scoring using additional basic scores and learning to rank}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure