TL;DR
This paper introduces a novel method for automatically assessing hypernasality in speech, a common symptom in motor-speech disorders like dysarthria. By leveraging acoustic models trained on healthy speech to capture nasal resonance and articulatory precision, their "Nasalization-Articulation Precision" (NAP) features achieve more robust and generalizable hypernasality estimation than existing methods, outperforming both engineered features and neural networks.
What Problem Does This Paper Address?
Hypernasality, characterized by excessive nasal resonance in speech, is a hallmark of velopharyngeal dysfunction (VPD) and a common symptom in various motor-speech disorders (e.g., Parkinson's, Huntington's disease, ALS, cleft palate). Accurately assessing hypernasality is crucial for tracking disease progression and speech therapy effectiveness. However, current methods have significant drawbacks:
- Clinician Perception: While the gold standard, it's subjective, prone to listener bias, and influenced by various factors (phonetic context, vocal quality), making it inconsistent and costly for large-scale tracking.
- Instrumentation-Based Methods: Techniques like videofluoroscopy and MRI are invasive and can be uncomfortable for patients. Nasometry, while non-invasive, shows only moderate correlation with perceptual judgment and requires specific test conditions.
- Existing Automated Methods:
- Engineered Features: Often fail to capture the complex acoustic patterns of hypernasality due to high speaker variability.
- Machine Learning Models: Tend to overfit small, disease-specific datasets, limiting their generalizability to new speakers or different disorders.
This paper aims to bridge this gap by developing an objective, robust, and generalizable automated method for hypernasality estimation.
Methods
The core of this work is the Nasalization-Articulation Precision (NAP) feature set. This approach combines domain expertise with machine learning, using two acoustic models trained exclusively on healthy speech:
- Nasalization Model (Voiced Phonemes): This model learns the acoustic patterns associated with nasal resonance in voiced phonemes. It uses a likelihood ratio derived from Gaussian Mixture Models (GMMs) trained on "nasal" (NAS) and "oral" (ORL) classes of voiced phonemes from healthy speech. A higher likelihood ratio for a voiced phoneme indicates increased nasalization.
- Articulation Model (Unvoiced Phonemes): This model captures the production quality of unvoiced phonemes, serving as a proxy for articulatory precision. It uses an acoustic model (trained with Kaldi) to assess how well unvoiced phonemes are articulated. A lower likelihood ratio (indicating less precise articulation) is expected with increased hypernasality.
These phoneme-level features (nasalization likelihood for voiced sounds and articulatory precision for unvoiced sounds) are then averaged per speaker and fed into a simple linear ridge regression model to predict clinician-assessed hypernasality scores. The key innovation is training the acoustic models on healthy speech, making them less susceptible to disease-specific confounds and enabling better generalization.
Datasets
- Healthy Speech Corpus: LibriSpeech dataset (approx. 1000 hours) was used to train the acoustic models.
- Dysarthric Speech Corpus: A dataset of 75 speakers (diagnosed with Parkinson's Disease, Huntington's Disease, Cerebellar Ataxia, and Amyotrophic Lateral Sclerosis) with clinician-rated hypernasality scores (1-7 scale) was used for training and evaluating the linear regression model.
- Cleft Palate Speech Corpus: A smaller corpus of 18 speakers (6 children, 12 adults) with cleft lip and palate was used to test the model's generalization to a different type of hypernasality without co-occurring neuromuscular diseases.
Key Insights and Ideas
- Healthy Speech as a Universal Baseline: Training acoustic models on healthy speech, rather than dysarthric speech, is crucial for creating features that are robust to disease-specific variations and generalize well.
- Complementary Features: Combining features that capture both nasal resonance (from voiced sounds) and articulatory imprecision (from unvoiced sounds) provides a more comprehensive picture of hypernasality.
- Articulatory Precision as a Proxy: For unvoiced phonemes, where direct nasal resonance is not the primary acoustic cue, assessing articulatory precision (i.e., how well the consonant is formed) serves as an effective indirect measure related to hypernasality.
- Interpretability: The NAP features are designed to be interpretable, with nasalization scores increasing and articulatory precision scores decreasing as hypernasality severity increases, aligning with perceptual expectations.
Key Results
- The NAP features, when used with a linear regression model, outperformed several state-of-the-art engineered features (formant-based) and neural network-based approaches (MFCC-NN, PASE-NN) in predicting clinician-assessed hypernasality scores.
- The model achieved better performance across most evaluation settings (LOSO and LODO cross-validation) compared to baselines, demonstrating strong generalization across different neurological disorders.
- In Leave-One-Disease-Out (LODO) evaluations, the NAP model showed significantly less accuracy loss than baseline methods, indicating its robustness to disease-specific manifestations.
- The model also outperformed human annotators in 8 out of 10 conditions, suggesting its potential as a more consistent and objective measure.
- When evaluated on the cleft palate speech corpus, the model trained on dysarthric speech achieved high correlation (PCC of 0.89 for adults, 0.82 for children), further supporting its ability to capture hypernasality independent of other neurological symptoms.
Limitations and Future Work
- Reliance on Forced Alignment: The current method requires accurate transcriptions for forced alignment, which can be challenging for spontaneous speech. While the model showed reasonable performance even with alignment errors, this remains a limitation.
- Unvoiced Phoneme Modeling: The nasalization model is trained on voiced phonemes. For unvoiced phonemes, articulatory precision is used as a proxy, which might not perfectly capture all aspects of nasalization.
- Correlation vs. Causation: While hypernasality and articulatory precision are correlated in dysarthria, the paper acknowledges the need for further work to fully disentangle the effects of hypernasality from general articulatory imprecision.
- Future Work:
- Develop neural models for direct assessment of articulatory precision and phoneme identification from raw audio, eliminating the need for transcripts.
- Collect a larger dataset of nasality-scored dysarthric speech across more diseases.
- Design stimuli better tailored for hypernasality assessment.
- Apply insights to improve the robustness of neural models for objective speech biomarkers.
Summary updated