TL;DR
This paper introduces an "Objective Articulation Measure" (OAM) using deep learning to analyze consonant-vowel (CV) transitions in speech. This OAM shows strong correlations with human perception of articulation quality across various speech disorders and accented speech, offering a more generalizable and computationally efficient alternative to existing methods.
What Problem Does This Paper Address?
Accurately assessing speech articulation is crucial for diagnosing and treating speech disorders, as well as understanding language acquisition and accent. Traditional methods often rely on subjective perceptual ratings by speech-language pathologists, which can be time-consuming and inconsistent. While acoustic analysis of speech has advanced, existing automated methods struggle with variability across different speech disorders (e.g., dysarthria, cleft palate) and accented speech. These variations in how articulatory impairments manifest acoustically make it difficult to develop a single, robust system that generalizes across these diverse populations. Specifically, the paper highlights that different error types produce distinct acoustic patterns in CV transitions, posing a challenge for disease- and speaker-invariant algorithms.
Methods
The core of this work is a deep learning model designed to extract features from the spectro-temporal dynamics of CV transition regions.
- CV Transition Segmentation: The system first segments CV transition regions around vowel onsets. A 160ms window (80ms before and 80ms after the vowel onset) is used, determined empirically to capture sufficient acoustic information for consonant discrimination.
- CNN Feature Extractor: A Convolutional Neural Network (CNN) is trained on a large corpus of healthy speech (LibriSpeech) to classify between 20 different consonants using mel-spectrograms of CV transition segments as input. This pre-trained CNN acts as a feature extractor.
- Objective Articulation Measure (OAM): The output of the CNN (specifically, the posterior probabilities of consonant classification) is used to compute the OAM. For a given CV transition, the OAM is calculated as the ratio of the posterior probability of the target consonant to the maximum posterior probability across all consonants. This score ranges from 0 to 1, with higher scores indicating better articulation.
- Speaker-Level OAM: For each speaker, consonant-level OAM scores are averaged to produce a single speaker-level OAM score.
- Linear Models: To further predict perceptual labels (e.g., articulation precision, acceptability, accentedness), linear models are trained using the speaker-level OAM scores.
Datasets
- Training Data: The LibriSpeech corpus, a large dataset of adult American English speech, was used to train the CNN consonant classifier.
- Evaluation Data: The OAM was evaluated on three distinct corpora:
- Dysarthria Database: Speech from 80 adults with varying levels of articulation impairment (Parkinson's disease, Huntington's disease, cerebellar Ataxia, ALS) and healthy controls. Perceptual ratings were provided by 14 speech-language pathologists.
- Utah Americleft Cleft Speech Database: Speech from 60 children (ages 5-7) with cleft lip/palate. Perceptual ratings were provided by 4 trained SLPs.
- Accented Speech Corpus: A subset of the GMU Speech Accent Archive, featuring native Mandarin and Spanish speakers of American English. Perceptual ratings were provided by 13 native American English speakers.
Key Insights and Ideas
- CV Transitions as Robust Articulation Cues: The paper reinforces the idea that CV transition regions contain invariant acoustic cues for articulation, making them suitable for objective assessment across different speakers and conditions.
- Generalizability of Deep Learning Features: A CNN trained on healthy speech can extract features that are surprisingly effective for evaluating articulation in diverse clinical populations and accented speech, even without specific fine-tuning on those populations. This "out-of-the-box" performance is a key contribution.
- OAM as a Unified Measure: The OAM provides a single, objective score that correlates well with human perceptual judgments, offering a consistent way to quantify articulation quality.
- Computational Efficiency: The OAM is significantly faster to compute than established methods like the Goodness of Pronunciation (GOP) algorithm, making it more practical for real-time applications or large-scale analysis.
- Saliency Maps Reveal Learned Features: Visualizing the CNN's internal workings with saliency maps shows that the model learns relevant acoustic features (like burst onsets and formant transitions) that correspond to articulatory events and errors.
Key Results
- Strong Correlations: Speaker-level averaged OAM scores showed significant correlations with perceptual ratings across all four evaluated databases:
- Dysarthria: r = -0.829 (OAM) vs. r = 0.836 (Linear Model)
- Utah Americleft: r = -0.659 (OAM) vs. r = 0.723 (Linear Model)
- Mandarin: r = -0.61 (OAM) vs. r = 0.812 (Linear Model)
- Spanish: r = -0.646 (OAM) vs. r = 0.788 (Linear Model) (Note: The negative correlation for OAM vs. Dysarthria is expected as higher OAM implies better articulation, while higher perceptual ratings in dysarthria indicate worse articulation.)
- Linear Model Improvement: Linear models built using OAM features further improved prediction accuracy compared to using the average OAM score alone.
- Reduced Variability: The OAM exhibited lower coefficient of variation compared to the GOP algorithm across all databases, indicating greater reliability and consistency.
- Computational Speed: The OAM computation was approximately 2X faster than the GOP algorithm.
Limitations and Future Work
- Dependency on Transcription and Alignment: A key limitation acknowledged is that both the OAM and GOP algorithms require accurate transcriptions and forced alignment to segment individual phonemes or CV transitions.
- Future Work: The authors suggest future research should focus on developing objective articulation assessment models that are independent of transcription and forced-alignment operations. This would make the system even more robust and applicable in scenarios where precise alignment is difficult or impossible.
Summary updated