TL;DR
This paper improves automatic pronunciation scoring by incorporating novel vowel- and consonant-specific word-level scores and applying learning-to-rank (LTR) algorithms. The approach significantly outperforms existing methods, demonstrating the value of finer-grained phonetic analysis and advanced machine learning for evaluating spoken language proficiency.
What Problem Does This Paper Address?
Automatic Pronunciation Scoring (APS) systems are crucial for Computer-Assisted Pronunciation Training (CAPT) by providing automated feedback on learners' spoken English. Existing APS systems often rely on a single type of "basic score" (e.g., HMM-based log-likelihood) or simple averaging of phone-level scores to derive a word-level score. This can overlook the distinct contributions of vowels and consonants to overall pronunciation quality. Furthermore, combining multiple basic scores effectively has been a challenge, with methods like neural networks being computationally expensive. This paper aims to address these limitations by exploring more informative word-level features and leveraging learning-to-rank techniques for more robust and efficient score combination.
Methods
The core of this work lies in two main areas:
- Enhanced Word-Level Scores: The authors extend their previous work by converting phone-level basic scores into word-level scores using three methods:
- Average-based: The standard method, averaging all phone scores within a word.
- Vowel-based: Averaging only the phone scores of vowel segments within a word.
-
Consonant-based: Averaging only the phone scores of consonant segments within a word. This creates 18 distinct word-level scores (6 basic scores x 3 conversion methods).
-
Learning to Rank (LTR): Instead of directly combining scores into a single output, LTR algorithms are used to learn a ranking function. This approach treats pronunciation scoring as an information retrieval problem, where the goal is to order utterances by their quality. The paper compares two LTR approaches:
- Pairwise (RankSVM): Learns by comparing pairs of utterances.
- Listwise (ListNet): Learns by considering the entire list of utterances at once.
The system also includes an LTR score quantization stage to convert continuous LTR scores into discrete ranks.
Datasets
- Native Corpus: The Wall Street Journal (WSJ) corpus was used to train Hidden Markov Models (HMMs) for speech recognition and to obtain necessary phonetic statistics.
- Non-Native Corpus: The MIR-SD dataset was used for training and evaluation. This dataset consists of 1100 utterances from 22 Taiwanese speakers, each speaking over 200 multi-syllabic English words. Human rankings on a scale of 1 (unintelligible) to 5 (native-like) were collected for this corpus, with reported word-based inter-rater correlation of 0.58 and speaker-based correlation of 0.78.
Key Insights and Ideas
- Vowel and Consonant Importance: The paper highlights that treating vowels and consonants separately when deriving word-level scores can capture nuances missed by simple averaging, leading to improved pronunciation assessment.
- Learning to Rank Efficacy: LTR methods, particularly ListNet, demonstrate superior performance compared to traditional score combination techniques like Gaussian classifiers or linear combinations.
- Computational Efficiency: The listwise LTR approach (ListNet) offers a significant advantage in terms of training time compared to the pairwise RankSVM, making it more practical for real-world applications.
- Feature Combination Benefits: Combining all three types of word-level scores (average, vowel, consonant) generally leads to better performance, especially when used with LTR methods.
Key Results
- The proposed LTR-based framework with enhanced word-level scores significantly outperforms most existing methods.
- Specifically, the ListNet approach, when using all 18 word-level scores, achieved the best performance across various evaluation metrics (raw correlation, rank correlation, rank recognition rate, etc.).
- The LTR methods (RankSVM and ListNet) performed better than traditional score combination methods like GCE, LCA, and NN.
- The best basic scoring method, HMM-based log-posterior probability scores (hmmPost), was surpassed by the LTR-based approaches.
- Training time for RankSVM was considerable (4-6 hours), while ListNet was much faster (a few seconds).
Limitations and Future Work
- Dataset Size: The authors acknowledge that the MIR-SD dataset, while novel, is relatively small. They plan to test their methods on a larger dataset currently under construction.
- LTR Algorithm Exploration: The paper focused on RankSVM and ListNet. Future work will explore other LTR algorithms, such as RBF-ListNet and RankNet.
- Generalizability: While the results are promising, further validation on diverse non-native speaker populations and different languages would be beneficial.
Summary updated