Introduction
Computer-Aided Pronunciation Training (CAPT) systems aim to provide automated, granular feedback to language learners. However, accurately grading individual sounds (phones) remains a significant challenge. Most industry-standard systems rely on the Goodness of Pronunciation (GOP) metric. While effective, GOP is typically derived from Automatic Speech Recognition (ASR) models trained exclusively on native speech. These models are designed to be robust to accents to maximize word recognition, making them "too permissive" for scoring—they often fail to penalize the specific phonetic errors made by non-native speakers.
Training specialized models from scratch is difficult due to the scarcity of phonetically labeled non-native speech datasets. This research proposes a transfer learning approach to bridge this gap, adapting native ASR models to the task of pronunciation scoring.
Methodology
The researchers developed a pipeline that leverages a pre-trained state-of-the-art ASR model (a Factorized Time Delay Neural Network, or TDNN-F) trained on 960 hours of native English.
The core of the approach involves:
- Architecture Modification: The final layer of the native ASR model, which predicts sub-phonetic states (senones), is replaced with a new layer dedicated to predicting the probability of a phone being correct or incorrect.
- Two-Stage Fine-Tuning: To prevent catastrophic forgetting of the acoustic features learned from the large native dataset, the model is trained in two steps. First, only the new output layer is trained. Subsequently, the second-to-last layer is unfrozen and trained alongside it.
- Balanced Loss and Cost Function: To address the class imbalance (as learners produce more correct than incorrect sounds), a balanced binary cross-entropy loss was used. Furthermore, the authors introduced a custom cost function ($0.5 \times \text{FPR} + \text{FNR}$) to prioritize reducing "false corrections," which are particularly discouraging for students.
Experimental Setup
The model was evaluated using EpaDB, a database containing 3,200 English utterances from 50 Spanish speakers, manually annotated at the phone level. The system's performance was compared against the standard GOP baseline.
Key Results
The transfer learning approach, specifically the configuration utilizing batch normalization, dropout, and balanced loss, significantly outperformed the GOP baseline across several metrics.
- Cost Reduction: The system achieved a 20% reduction in the custom cost function on the evaluation set.
- Discriminative Power: There was a 40% improvement in 1-AUC, indicating a much higher ability to distinguish between correct and incorrect pronunciations.
- Error Detection: The False Positive Rate (FPR) dropped from 64% (GOP) to 41%, meaning the system caught significantly more actual errors without increasing the rate of false alarms.
Discussion and Conclusion
The results demonstrate that fine-tuning native ASR models "tightens" the phonetic boundaries, making them more suitable for the rigorous demands of pronunciation scoring. The study also highlights the importance of careful unfreezing; while fine-tuning the last two layers improved performance, unfreezing deeper layers led to overfitting on the small non-native dataset.
While the system shows marked improvements, performance varies across different phones. Future work will focus on expanding the training data to include more diverse native language (L1) backgrounds and exploring larger foundation models like Wav2Vec 2.0 to further enhance scoring accuracy.
Summary updated