A comparison of GMM-HMM and DNN-HMM based pronunciation verification techniques for use in the assessment of childhood apraxia of speech

Mostafa Shahin, Beena Ahmed, Jacqueline McKechnie, Kirrie Ballard, Ricardo Gutierrez-OsunaInterspeech 2014

TL;DR

This paper introduces a novel pronunciation verification system for assessing childhood apraxia of speech (CAS), a developmental disorder affecting speech articulation. By comparing traditional GMM-HMM models with newer DNN-HMMs, the authors demonstrate that DNN-HMMs significantly improve accuracy in identifying pronunciation errors, paving the way for more effective automated speech therapy tools.

What Problem Does This Paper Address?

Childhood apraxia of speech (CAS) is a motor speech disorder that makes it difficult for children to produce sounds and syllables correctly. Traditional therapy for CAS involves intensive sessions with speech-language pathologists (SLPs), which can be costly and logistically challenging. While automated tools could offer a more accessible and adaptive solution, existing systems often lack accurate, real-time feedback on pronunciation. This paper tackles the challenge of developing a robust pronunciation verification system that can accurately identify mispronounced phonemes in children with CAS, a task made difficult by the inherent variability and error patterns in their speech.

Methods

The core of the proposed system is a pronunciation verification module designed to compare a child's speech production against a target pronunciation. This is achieved by:

  • Lattice Creation: A flexible search lattice is generated for each target word. This lattice includes not only the correct phoneme sequence but also alternative phonemes representing common mispronunciations in CAS, as identified by SLPs. A "garbage model" is also incorporated to capture unexpected inserted sounds. Penalties are applied to alternative and garbage paths to control the system's strictness.
  • Acoustic Modeling: The system compares two types of acoustic models:
  • GMM-HMM (Gaussian Mixture Model - Hidden Markov Model): A conventional approach for speech recognition.
  • DNN-HMM (Deep Neural Network - Hidden Markov Model): A hybrid model that leverages the power of deep learning. DNN-HMMs are known to perform well, especially with limited training data.
  • Speech Recognition: The child's speech signal is processed, features are extracted, and then fed into a speech recognizer along with the created lattice and the chosen acoustic model. The recognizer outputs a phoneme sequence.
  • Evaluation: The recognized phoneme sequence is compared to the correct phoneme sequence to identify and classify errors (insertion, deletion, substitution).

Datasets

The study utilizes two speech corpora:

  • OGI Kids' Speech Corpus: A large corpus of normal speech from children aged 5-10 years, containing isolated words, sentences, and numeric strings. For this study, only "good" utterances of isolated words were used.
  • Disordered Speech Corpus: A custom corpus collected from 41 children (aged 4-12) diagnosed with CAS. This corpus consists of 90 isolated words per child, recorded in a clinical setting and phonetically annotated by SLPs.

Key Insights and Ideas

  • DNN-HMMs Outperform GMM-HMMs for Disordered Speech: The most significant finding is the superior performance of DNN-HMMs over GMM-HMMs, particularly for the challenging task of recognizing disordered speech. This highlights the advantage of deep learning models in capturing complex and variable speech patterns.
  • Importance of Contextualized Mispronunciation Rules: The use of rules derived from SLP assessments to create a flexible search lattice, incorporating expected mispronunciations, is crucial for accurately verifying pronunciation in CAS.
  • Trade-off in Acoustic Model Tuning: For GMM-HMMs, the number of mixtures per state needs careful tuning. Too many mixtures can lead to overfitting and poor performance with limited data, as seen with the disordered speech corpus.
  • Impact of Input Window Size for DNNs: For DNN-HMMs, increasing the input window size improved performance up to a certain point (590ms for normal, 470ms for disordered speech). This suggests that longer temporal contexts are beneficial for modeling phonemes, especially in therapy settings where word sets are limited.

Key Results

  • Overall Accuracy: The DNN-HMM system achieved a total phoneme level accuracy of 89% for disordered speech, which is comparable to inter-rater reliability among SLPs.
  • Error Detection: The system could detect correctly pronounced phonemes with 94% accuracy and identify specific mispronunciation errors with 54% accuracy.
  • Phoneme Error Rate (PER):
    • GMM-HMM: Best PER of 37% (normal speech) and 43.4% (disordered speech).
  • DNN-HMM: Best PER of 21% (normal speech) and 36.2% (disordered speech).
  • Improvement with DNN-HMM: The DNN-HMM demonstrated a 27.2% improvement in detecting mispronounced phonemes and a 1.9% improvement in correctly identifying pronounced phonemes compared to GMM-HMMs for disordered speech.

Limitations and Future Work

  • Limited Data for Disordered Speech: The disordered speech corpus, while valuable, is smaller than the normal speech corpus, which can impact model training. The DNN-HMM's ability to perform better with limited data is a key advantage.
  • Comparison to Existing Systems: The authors note that direct comparison to existing automated ASR systems for CAS is difficult due to the novelty of this specific application.
  • Future Work: The paper suggests that the developed system can be integrated into an automated speech therapy tool to provide accurate classification of mispronunciation errors in noisy clinical environments. Further research could explore more advanced deep learning architectures and larger, more diverse datasets.

Summary updated

BibTeX
@inproceedings{shahin2014comparison,
  author = {Mostafa Shahin and Beena Ahmed and Jacqueline McKechnie and Kirrie Ballard and Ricardo Gutierrez-Osuna},
  title = {{A comparison of GMM-HMM and DNN-HMM based pronunciation verification techniques for use in the assessment of childhood apraxia of speech}},
  booktitle = {Interspeech},
  year = {2014},
}
Figure