A Transfer Learning Approach for Pronunciation Scoring

Marcelo Sancinetti, Jazmín Vidal, Cyntia Bonomi, Luciana Ferrer

Introduction

Computer-Aided Pronunciation Training (CAPT) systems aim to provide automated, granular feedback to language learners. However, accurately grading individual sounds (phones) remains a significant challenge. Most industry-standard systems rely on the Goodness of Pronunciation (GOP) metric. While effective, GOP is typically derived from Automatic Speech Recognition (ASR) models trained exclusively on native speech. These models are designed to be robust to accents to maximize word recognition, making them "too permissive" for scoring—they often fail to penalize the specific phonetic errors made by non-native speakers.

Training specialized models from scratch is difficult due to the scarcity of phonetically labeled non-native speech datasets. This research proposes a transfer learning approach to bridge this gap, adapting native ASR models to the task of pronunciation scoring.

Methodology

The researchers developed a pipeline that leverages a pre-trained state-of-the-art ASR model (a Factorized Time Delay Neural Network, or TDNN-F) trained on 960 hours of native English.

The proposed DNN architecture and scoring pipeline, showing the transition from native ASR to a phone-level scoring model.

The core of the approach involves:

  1. Architecture Modification: The final layer of the native ASR model, which predicts sub-phonetic states (senones), is replaced with a new layer dedicated to predicting the probability of a phone being correct or incorrect.
  2. Two-Stage Fine-Tuning: To prevent catastrophic forgetting of the acoustic features learned from the large native dataset, the model is trained in two steps. First, only the new output layer is trained. Subsequently, the second-to-last layer is unfrozen and trained alongside it.
  3. Balanced Loss and Cost Function: To address the class imbalance (as learners produce more correct than incorrect sounds), a balanced binary cross-entropy loss was used. Furthermore, the authors introduced a custom cost function ($0.5 \times \text{FPR} + \text{FNR}$) to prioritize reducing "false corrections," which are particularly discouraging for students.

Experimental Setup

The model was evaluated using EpaDB, a database containing 3,200 English utterances from 50 Spanish speakers, manually annotated at the phone level. The system's performance was compared against the standard GOP baseline.

Key Results

The transfer learning approach, specifically the configuration utilizing batch normalization, dropout, and balanced loss, significantly outperformed the GOP baseline across several metrics.

Comparison of the proposed fine-tuned model (GOP-FT) against the standard GOP baseline across various English phones.
  • Cost Reduction: The system achieved a 20% reduction in the custom cost function on the evaluation set.
  • Discriminative Power: There was a 40% improvement in 1-AUC, indicating a much higher ability to distinguish between correct and incorrect pronunciations.
  • Error Detection: The False Positive Rate (FPR) dropped from 64% (GOP) to 41%, meaning the system caught significantly more actual errors without increasing the rate of false alarms.

Discussion and Conclusion

The results demonstrate that fine-tuning native ASR models "tightens" the phonetic boundaries, making them more suitable for the rigorous demands of pronunciation scoring. The study also highlights the importance of careful unfreezing; while fine-tuning the last two layers improved performance, unfreezing deeper layers led to overfitting on the small non-native dataset.

While the system shows marked improvements, performance varies across different phones. Future work will focus on expanding the training data to include more diverse native language (L1) backgrounds and exploring larger foundation models like Wav2Vec 2.0 to further enhance scoring accuracy.

Summary updated

Abstract

Phone-level pronunciation scoring is a challenging task, with performance far from that of human annotators. Standard systems generate a score for each phone in a phrase using models trained for automatic speech recognition (ASR) with native data only. Better performance has been shown when using systems that are trained specifically for the task using nonnative data. Yet, such systems face the challenge that datasets labelled for this task are scarce and usually small. In this paper, we present a transfer learning-based approach that leverages a model trained for ASR, adapting it for the task of pronunciation scoring. We analyze the effect of several design choices and compare the performance with a state-of-the-art goodness of pronunciation (GOP) system. Our final system is 20% better than the GOP system on EpaDB, a database for pronunciation scoring research, for a cost function that prioritizes low rates of unnecessary corrections.
BibTeX
@inproceedings{sancinetti2022transfer,
  author = {Marcelo Sancinetti and Jazmín Vidal and Cyntia Bonomi and Luciana Ferrer},
  title = {{A Transfer Learning Approach for Pronunciation Scoring}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=m45WGyF56p},
}
Figure