Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment

Kwanghee Choi, Eunjung Yeo, Kalvin Chang, Shinji Watanabe, David R. Mortensen

Abstract

Allophony refers to the variation in the phonetic realization of a phoneme based on its phonetic environment. Modeling allophones is crucial for atypical pronunciation assessment, which involves distinguishing atypical from typical pronunciations. However, recent phoneme classifier-based approaches often simplify this by treating various realizations as a single phoneme, bypassing the complexity of modeling allophonic variation. Motivated by the acoustic modeling capabilities of frozen self-supervised speech model (S3M) features, we propose MixGoP, a novel approach that leverages Gaussian mixture models to model phoneme distributions with multiple subclusters. Our experiments show that MixGoP achieves state-of-the-art performance across four out of five datasets, including dysarthric and non-native speech. Our analysis further suggests that S3M features capture allophonic variation more effectively than MFCCs and Mel spectrograms, highlighting the benefits of integrating MixGoP with S3M features.

Introduction

In speech therapy and language learning, the Goodness of Pronunciation (GoP) metric is widely used to quantify how "correct" a speaker sounds. Traditionally, GoP relies on phoneme classifiers that treat every instance of a phoneme (e.g., /t/) as a single, uniform target. However, human language exhibits significant phonetic diversity through allophones—subtle variations in how a phoneme is realized depending on its phonetic context. For instance, the /t/ in "tap" is aspirated, while in "stop" it is unaspirated, and in "butter" it often becomes a flap [ɾ].

Standard deep learning models often "squash" these variations into a single cluster to simplify classification. This leads to two major issues in atypical speech assessment:

  1. Unimodal Simplification: Treating a phoneme as one cluster ignores valid phonetic diversity, resulting in inaccurate atypicality scores.
  2. In-Distribution Bias: Classifiers trained on typical speech often produce unreliable softmax probabilities when encountering "out-of-distribution" (OOD) atypical speech, such as that from individuals with dysarthria.

Methods: Modeling Allophonic Variation with MixGoP

To address these limitations, we introduce MixGoP, a method that shifts from simple classification toward likelihood modeling using Gaussian Mixture Models (GMMs).

Conceptual illustration of allophonic variation and likelihood-based atypicality detection.

Core Components

  • Gaussian Mixture Models (GMMs): Instead of a single centroid per phoneme, MixGoP employs a GMM with multiple subclusters (typically 32). This allows the model to explicitly represent different allophones as distinct peaks in the distribution.
  • Self-Supervised Speech Models (S3Ms): We utilize features from frozen pre-trained models like WavLM and XLS-R. These models, trained on vast unlabeled datasets, are known to capture rich phonetic context.
  • Likelihood vs. Posterior: While standard GoP uses posterior probability, MixGoP calculates the log-likelihood—measuring how well a sound fits the known distribution of a phoneme. By avoiding the softmax function, the model becomes significantly more robust to OOD atypical speech.

Experimental Setup

We evaluated MixGoP across five diverse datasets to ensure robustness across different types of speech atypicality:

  • Dysarthric Speech: UASpeech (English), TORGO (English), and SSNCE (Tamil).
  • Non-native Speech: speechocean762 (Mandarin speakers learning English) and L2-ARCTIC (various L1 backgrounds).

The models were trained on typical (healthy/native) speech and tested on their respective atypical counterparts.

Results and Analysis

MixGoP achieved state-of-the-art performance, showing the highest correlation with human-labeled pronunciation scores in 4 out of 5 datasets.

Performance comparison (Kendall-tau correlation) of MixGoP against baseline methods across five datasets.

Key Insights

  • S3Ms are Allophone-Aware: Analysis revealed that S3M features (particularly in middle layers) naturally cluster by phonetic environment. For example, the phoneme /AH/ splits into sub-clusters based on whether the adjacent sound is a vowel or a consonant.
  • OOD Robustness: MixGoP performed exceptionally well on dysarthric speech. Because dysarthric speech is highly OOD compared to healthy speech, modeling the exact likelihood of sub-clusters is far more effective than using a broad classifier.
  • Feature Superiority: WavLM and XLS-R features vastly outperformed traditional MFCCs and Mel-spectrograms, confirming that S3Ms store critical "relative distance" information necessary for atypicality detection.
  • Layer Sensitivity: Performance varies by layer; for XLS-R, the middle layers (around layer 19) were optimal, while WavLM maintained strong performance into its final layers.

Conclusion and Future Work

MixGoP demonstrates that modeling allophonic variation through GMMs and S3M features significantly improves atypical pronunciation assessment. While highly effective, the method currently relies on forced alignment, which can be challenging for severely atypical speech. Future work will explore broader cross-linguistic validation beyond English and Tamil, and investigate more sophisticated weighting mechanisms to prioritize phonemes that most impact overall speech intelligibility.

Summary updated

Abstract

Allophony refers to the variation in the phonetic realization of a phoneme based on its phonetic environment. Modeling allophones is crucial for atypical pronunciation assessment, which involves distinguishing atypical from typical pronunciations. However, recent phoneme classifier-based approaches often simplify this by treating various realizations as a single phoneme, bypassing the complexity of modeling allophonic variation. Motivated by the acoustic modeling capabilities of frozen self-supervised speech model (S3M) features, we propose MixGoP, a novel approach that leverages Gaussian mixture models to model phoneme distributions with multiple subclusters. Our experiments show that MixGoP achieves state-of-the-art performance across four out of five datasets, including dysarthric and non-native speech. Our analysis further suggests that S3M features capture allophonic variation more effectively than MFCCs and Mel spectrograms, highlighting the benefits of integrating MixGoP with S3M features.
BibTeX
@inproceedings{choi2025leveraging,
  author = {Kwanghee Choi and Eunjung Yeo and Kalvin Chang and Shinji Watanabe and David R. Mortensen},
  title = {{Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.132},
}
Figure