Robust Estimation of Hypernasality in Dysarthria With Acoustic Model Likelihood Features

Michael Saxon, Ayush Tripathi, Yishan Jiao, Julie M. Liss, Visar Berisha

TL;DR

This paper introduces a novel method for automatically assessing hypernasality in speech, a common symptom in motor-speech disorders like dysarthria. By leveraging acoustic models trained on healthy speech to capture nasal resonance and articulatory precision, their "Nasalization-Articulation Precision" (NAP) features achieve more robust and generalizable hypernasality estimation than existing methods, outperforming both engineered features and neural networks.

What Problem Does This Paper Address?

Hypernasality, characterized by excessive nasal resonance in speech, is a hallmark of velopharyngeal dysfunction (VPD) and a common symptom in various motor-speech disorders (e.g., Parkinson's, Huntington's disease, ALS, cleft palate). Accurately assessing hypernasality is crucial for tracking disease progression and speech therapy effectiveness. However, current methods have significant drawbacks:

  • Clinician Perception: While the gold standard, it's subjective, prone to listener bias, and influenced by various factors (phonetic context, vocal quality), making it inconsistent and costly for large-scale tracking.
  • Instrumentation-Based Methods: Techniques like videofluoroscopy and MRI are invasive and can be uncomfortable for patients. Nasometry, while non-invasive, shows only moderate correlation with perceptual judgment and requires specific test conditions.
  • Existing Automated Methods:
    • Engineered Features: Often fail to capture the complex acoustic patterns of hypernasality due to high speaker variability.
  • Machine Learning Models: Tend to overfit small, disease-specific datasets, limiting their generalizability to new speakers or different disorders.

This paper aims to bridge this gap by developing an objective, robust, and generalizable automated method for hypernasality estimation.

Methods

The core of this work is the Nasalization-Articulation Precision (NAP) feature set. This approach combines domain expertise with machine learning, using two acoustic models trained exclusively on healthy speech:

  1. Nasalization Model (Voiced Phonemes): This model learns the acoustic patterns associated with nasal resonance in voiced phonemes. It uses a likelihood ratio derived from Gaussian Mixture Models (GMMs) trained on "nasal" (NAS) and "oral" (ORL) classes of voiced phonemes from healthy speech. A higher likelihood ratio for a voiced phoneme indicates increased nasalization.
  2. Articulation Model (Unvoiced Phonemes): This model captures the production quality of unvoiced phonemes, serving as a proxy for articulatory precision. It uses an acoustic model (trained with Kaldi) to assess how well unvoiced phonemes are articulated. A lower likelihood ratio (indicating less precise articulation) is expected with increased hypernasality.

These phoneme-level features (nasalization likelihood for voiced sounds and articulatory precision for unvoiced sounds) are then averaged per speaker and fed into a simple linear ridge regression model to predict clinician-assessed hypernasality scores. The key innovation is training the acoustic models on healthy speech, making them less susceptible to disease-specific confounds and enabling better generalization.

Datasets

  • Healthy Speech Corpus: LibriSpeech dataset (approx. 1000 hours) was used to train the acoustic models.
  • Dysarthric Speech Corpus: A dataset of 75 speakers (diagnosed with Parkinson's Disease, Huntington's Disease, Cerebellar Ataxia, and Amyotrophic Lateral Sclerosis) with clinician-rated hypernasality scores (1-7 scale) was used for training and evaluating the linear regression model.
  • Cleft Palate Speech Corpus: A smaller corpus of 18 speakers (6 children, 12 adults) with cleft lip and palate was used to test the model's generalization to a different type of hypernasality without co-occurring neuromuscular diseases.

Key Insights and Ideas

  • Healthy Speech as a Universal Baseline: Training acoustic models on healthy speech, rather than dysarthric speech, is crucial for creating features that are robust to disease-specific variations and generalize well.
  • Complementary Features: Combining features that capture both nasal resonance (from voiced sounds) and articulatory imprecision (from unvoiced sounds) provides a more comprehensive picture of hypernasality.
  • Articulatory Precision as a Proxy: For unvoiced phonemes, where direct nasal resonance is not the primary acoustic cue, assessing articulatory precision (i.e., how well the consonant is formed) serves as an effective indirect measure related to hypernasality.
  • Interpretability: The NAP features are designed to be interpretable, with nasalization scores increasing and articulatory precision scores decreasing as hypernasality severity increases, aligning with perceptual expectations.

Key Results

  • The NAP features, when used with a linear regression model, outperformed several state-of-the-art engineered features (formant-based) and neural network-based approaches (MFCC-NN, PASE-NN) in predicting clinician-assessed hypernasality scores.
  • The model achieved better performance across most evaluation settings (LOSO and LODO cross-validation) compared to baselines, demonstrating strong generalization across different neurological disorders.
  • In Leave-One-Disease-Out (LODO) evaluations, the NAP model showed significantly less accuracy loss than baseline methods, indicating its robustness to disease-specific manifestations.
  • The model also outperformed human annotators in 8 out of 10 conditions, suggesting its potential as a more consistent and objective measure.
  • When evaluated on the cleft palate speech corpus, the model trained on dysarthric speech achieved high correlation (PCC of 0.89 for adults, 0.82 for children), further supporting its ability to capture hypernasality independent of other neurological symptoms.

Limitations and Future Work

  • Reliance on Forced Alignment: The current method requires accurate transcriptions for forced alignment, which can be challenging for spontaneous speech. While the model showed reasonable performance even with alignment errors, this remains a limitation.
  • Unvoiced Phoneme Modeling: The nasalization model is trained on voiced phonemes. For unvoiced phonemes, articulatory precision is used as a proxy, which might not perfectly capture all aspects of nasalization.
  • Correlation vs. Causation: While hypernasality and articulatory precision are correlated in dysarthria, the paper acknowledges the need for further work to fully disentangle the effects of hypernasality from general articulatory imprecision.
  • Future Work:
    • Develop neural models for direct assessment of articulatory precision and phoneme identification from raw audio, eliminating the need for transcripts.
  • Collect a larger dataset of nasality-scored dysarthric speech across more diseases.
  • Design stimuli better tailored for hypernasality assessment.
  • Apply insights to improve the robustness of neural models for objective speech biomarkers.

Summary updated

Abstract

Hypernasality is a common characteristic symptom across many motor-speech disorders. For voiced sounds, hypernasality introduces an additional resonance in the lower frequencies and, for unvoiced sounds, there is reduced articulatory precision due to air escaping through the nasal cavity. However, the acoustic manifestation of these symptoms is highly variable, making hypernasality estimation very challenging, both for human specialists and automated systems. Previous work in this area relies on either engineered features based on statistical signal processing or machine learning models trained on clinical ratings. Engineered features often fail to capture the complex acoustic patterns associated with hypernasality, whereas metrics based on machine learning are prone to overfitting to the small disease-specific speech datasets on which they are trained. Here we propose a new set of acoustic features that capture these complementary dimensions. The features are based on two acoustic models trained on a large corpus of healthy speech. The first acoustic model aims to measure nasal resonance from voiced sounds, whereas the second acoustic model aims to measure articulatory imprecision from unvoiced sounds. To demonstrate that the features derived from these acoustic models are specific to hypernasal speech, we evaluate them across different dysarthria corpora. Our results show that the features generalize even when training on hypernasal speech from one disease and evaluating on hypernasal speech from another disease (e.g. training on Parkinson's disease, evaluation on Huntington's disease), and when training on neurologically disordered speech but evaluating on cleft palate speech.
BibTeX
@article{saxon2020robust,
  author = {Michael Saxon and Ayush Tripathi and Yishan Jiao and Julie M. Liss and Visar Berisha},
  title = {{Robust Estimation of Hypernasality in Dysarthria With Acoustic Model Likelihood Features}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=o2C_zGsVWF},
}
Figure