Pronunciation quality evaluation of sentences by combining word based scores

Jorge Wuth, Néstor Becerra Yoma, Leopoldo Benavides, Hiram VivancoInterspeech 2012

TL;DR

This paper tackles the challenge of automatically evaluating the pronunciation quality of entire sentences for second language learners. They propose a novel method that combines word-level objective scores, generated using an ASR system with a competitive vocabulary and phonetic variants, to achieve sentence-level evaluations that correlate well with human judgment. This approach significantly reduces the need for extensive prior studies of common pronunciation errors.

What Problem Does This Paper Address?

Evaluating pronunciation quality in second language learning (2LL) is crucial for effective computer-aided pronunciation training (CAPT). While evaluating individual words has seen progress, assessing the pronunciation of full sentences remains complex. Human evaluators consider various phonetic and suprasegmental aspects, making automatic evaluation difficult. Existing automatic methods often rely on sub-word units and simple averaging of scores, or require extensive prior knowledge of common student mistakes and specific target utterances. This paper aims to bridge this gap by developing a more robust and less labor-intensive method for sentence-level pronunciation quality assessment that doesn't require pre-defined error profiles.

Methods

The core of the proposed method lies in combining word-based objective scores to derive a sentence-level score. Here's a breakdown of their approach:

  • Subjective Criterion: The authors define a subjective sentence score as the average of subjective scores assigned to individual words within the sentence.
  • Objective Sentence Score: Correspondingly, the objective sentence score is derived by averaging objective scores calculated for each word in the sentence.
  • ASR-Based Objective Metric: They employ an Automatic Speech Recognition (ASR) system with a class-based language model. This system is designed to handle "competitive classes" for each target word.
  • Competitive Class Generation: For each target word, a competitive class is created, comprising:
  • The target word itself with its correct pronunciation.
  • A competitive lexicon of similar-sounding words.
  • Phonetic variants of the target word, generated by applying phonetic rules of the student's native language (Spanish in this case) and mapping them to similar English phonemes. This is a key innovation as it avoids the need for explicit error studies.
  • Word-based N-best List Feature Extraction: The ASR system outputs several confidence measures for each word, including N-best position, recognition flag, word density confidence measure, and logarithmic word density confidence measure.
  • Word-based Objective Pronunciation Score Estimation: These confidence measures are then used to estimate an objective pronunciation score for each word. They employ multi-classifier system (MCS) techniques, specifically the mean and majority vote rules, to map these confidence metrics to a score that emulates subjective human judgment.
  • Class-Based Language Model: A class-based language model is used within the ASR system to identify potential mispronounced words by considering the probabilities of these competitive classes.

Datasets

  • Training Data: Native American English acoustic models were trained using the CSR-I WSJO corpus.
  • Evaluation Data: A custom database was created, consisting of 423 utterances from 43 speakers with varying English proficiency levels. These utterances were extracted from a web-based 2LL system and selected by an expert for phonetic balance. The database was split into two subsets for training and testing, with experiments conducted in both directions.

Key Insights and Ideas

  • Sentence-level evaluation by averaging word-scores: The paper demonstrates that averaging word-level objective scores can effectively approximate a subjective sentence-level evaluation criterion. This simplifies the complex task of sentence assessment.
  • Automatic generation of phonetic variants: The ability to automatically generate phonetic variants based on the student's native language (Spanish) and a predefined mapping (Table 1) is a significant contribution. This bypasses the laborious process of creating detailed error profiles for each target utterance.
  • Competitive vocabulary and phonetic variants enhance ASR for CAPT: By forcing the ASR system to consider not just the correct pronunciation but also plausible mispronunciations (via competitive lexicon and phonetic variants), the system becomes more adept at identifying subtle pronunciation errors.
  • Trade-off in competitive class complexity: The authors acknowledge that increasing the number of competing words and phonetic variants in a class can improve discrimination but also makes the ASR task more difficult, highlighting a practical design consideration.

Key Results

  • The proposed method achieved subjective-objective sentence score correlations as high as 0.5.
  • This correlation is considered very comparable to results reported in other studies that do require a priori studies of pronunciation errors.
  • The inter-annotator correlation for human subjective scores was 0.65, indicating a reasonable level of agreement among human evaluators.
  • Using a threshold (DPVmin) for phonetic variants in the competitive class led to improvements in the subjective-objective score correlation, with the maximum correlation achieved at DPVmin = 1. Specifically, DPVmin = 1 introduced increases of 11.1% and 8.3% in correlation for MCS mean and majority vote rules, respectively, compared to a case with no threshold.

Limitations and Future Work

  • Subjective-Objective Correlation: While the achieved correlation of 0.5 is good, it's still lower than the inter-annotator correlation of 0.65, suggesting room for further improvement in emulating human judgment.
  • Computational Complexity: The generation of competitive classes and the use of ASR with these classes can be computationally intensive, although the paper focuses on the evaluation accuracy.
  • Future Work: The paper implicitly suggests that further refinement of the competitive class generation, the MCS techniques, and potentially exploring other objective metrics could lead to even higher correlations. The efficiency and reduced human assistance offered by this approach also point towards its practical application in developing more dynamic CAPT systems.

Summary updated

BibTeX
@inproceedings{wuth2012pronunciation,
  author = {Jorge Wuth and Néstor Becerra Yoma and Leopoldo Benavides and Hiram Vivanco},
  title = {{Pronunciation quality evaluation of sentences by combining word based scores}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure