Transparent Pronunciation Scoring Using Articulatorily Weighted Phoneme Edit Distance

Reima Karhila, Anna-Riikka Smolander, Sari Ylinen, Mikko KurimoInterspeech 2019

TL;DR

This paper introduces a novel, transparent pronunciation scoring system for language learning games. By leveraging articulatory features to weight phoneme edit distances, their method provides interpretable feedback and achieves performance comparable to black-box approaches, paving the way for more effective Computer Assisted Pronunciation Training (CAPT).

What Problem Does This Paper Address?

Learning a new language, especially for children, often involves mastering pronunciation. Computer Assisted Pronunciation Training (CAPT) systems aim to provide learners with feedback on their spoken output. Traditional methods often rely on "black-box" acoustic models that can be difficult to interpret, making it hard for learners to understand why they made a mistake. Furthermore, generating detailed phonetic transcriptions of pronunciation errors for L2 learners is resource-intensive. This paper addresses the need for a CAPT scoring system that is not only accurate but also provides transparent, human-readable feedback on pronunciation errors, while also being robust to noisy annotations and requiring fewer resources for training.

Methods

The core of the proposed system is an articulatorily weighted phoneme edit distance metric. Instead of treating all phoneme substitutions, insertions, and deletions equally (as in standard Levenshtein distance), this method assigns costs based on the articulatory and phonological differences between phones.

Here's a breakdown of the key techniques:

  • Phoneme Recognition: A multilingual Connectionist Temporal Classification (CTC) model is used for phoneme recognition. CTC is chosen for its ability to handle unsegmented speech and its end-to-end training capabilities, making it suitable for this task.
  • Articulatory Feature Extraction: Phones are mapped to feature vectors representing articulatory characteristics (e.g., place and manner of articulation, voicing). These features are derived from a comprehensive set of descriptors.
  • Data-Driven Phonologically Weighted Levenshtein Distance (DD-PWLD): The system learns optimal weights for these articulatory features from a corpus of scored L2 utterances. This allows the model to adapt to specific learner populations and error patterns.
  • Mapping to a Score: The computed edit distance is then mapped to a user-friendly 0-5 star score using a tanh function, ensuring the feedback is usable in a gaming context.
  • Transparency: The system can break down the total score into contributions from individual phoneme errors, providing a human-readable list of mispronunciations and their impact.

Datasets

The experiments utilized a combination of datasets:

  • Native Speech Data:
    • WSJCAMO (adults) and PF-STAR (children) for UK English.
  • WSJ1 (adults) and TI-DIGITS (children) for American English.
  • SPEECON and SPEECHDAT (adults and children) for Finnish.
  • Språkbanken for Swedish.
  • L2 Learner Data: Speech data from Finnish 10-12 year old children learning UK English, collected using the game prototype.
  • Annotation: Human annotations were provided by a single language teacher with L1 Finnish, rating utterances on a 0-100 scale, later mapped to a 0-5 scale.

Key Insights and Ideas

  • Articulatory Features Enhance Scoring: Using articulatory features to weight phoneme errors provides a more nuanced and linguistically grounded measure of pronunciation quality than simple edit distance.
  • Data-Driven Weighting is Crucial: Optimizing these weights from actual learner data (DD-PWLD) significantly improves scoring accuracy compared to using default phonological weights.
  • Transparency Aids Learning: The ability to pinpoint specific phoneme errors and their contribution to the overall score offers valuable, interpretable feedback for learners, unlike opaque black-box models.
  • Robustness to Noise: The system is designed to be robust to noisy annotations, a common issue in collecting large-scale L2 speech data.
  • CTC as a Foundation: The CTC framework proves effective for multilingual phoneme recognition, providing a solid base for subsequent scoring.

Key Results

  • The DD-PWLD approach, when trained on L2 learner data, improved scoring performance by approximately 50% compared to standard PWLD.
  • The proposed DD-PWLD method achieved correlation coefficients with human annotations comparable to established black-box regression methods (SVM, RF), reaching up to 0.54 with outliers removed.
  • While black-box methods still slightly outperformed DD-PWLD, the proposed method offers the significant advantage of interpretability.
  • The system can provide a breakdown of scores for individual phoneme errors, making the feedback transparent and actionable.

Limitations and Future Work

  • Recognizer Performance: The current phoneme recognition accuracy (around 50% PER for native data) is a significant limitation, as the scoring system's performance is directly tied to the recognizer's quality.
  • Phoneme Set Merging: The phoneme set used across languages is still evolving, and further refinement is expected to improve performance.
  • Code-Switching: The CTC model struggles with mid-utterance code-switching, which is common in L2 learners. Future work will focus on improving this capability.
  • Data Variability: The quality of the collected L2 learner data is variable, and the system needs to be further optimized for such conditions.
  • Comparison to Other Methods: Future work will involve a more in-depth comparison with other established pronunciation quality assessment methods, such as Dynamic Time Warping (DTW) and forced alignment.

Summary updated

BibTeX
@inproceedings{karhila2019transparent,
  author = {Reima Karhila and Anna-Riikka Smolander and Sari Ylinen and Mikko Kurimo},
  title = {{Transparent Pronunciation Scoring Using Articulatorily Weighted Phoneme Edit Distance}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure