Abstract
Dysarthric speech recognition (DSR) is a critical task for improving accessibility, yet it remains challenging due to the high variability of speech impairments and the significant acoustic gap between dysarthric and healthy speech. We introduce DyPCL (Dynamic Phoneme-level Contrastive Learning), a novel framework designed to enhance DSR by focusing on fine-grained phonetic representations. DyPCL utilizes a dynamic alignment mechanism to extract speech embeddings directly from the Connectionist Temporal Classification (CTC) layer and employs a curriculum-based contrastive learning strategy. By progressively introducing phonetically similar "hard" negatives, the model learns to distinguish subtle mispronunciations. Experimental results on the UASpeech dataset demonstrate that DyPCL achieves a 22.1% relative reduction in Word Error Rate (WER) across all severity levels, with particularly significant gains for speakers with very low intelligibility.
Introduction
Dysarthria is a motor speech disorder resulting from neurological injury, characterized by slurred, slow, or irregular speech. These characteristics pose two primary challenges for standard Automatic Speech Recognition (ASR) systems: intrinsic diversity (symptoms vary significantly between individuals) and extrinsic disparity (impaired speech deviates sharply from the "healthy" data used to train most models).
Previous approaches to DSR have relied on data augmentation or word-level contrastive learning. However, word-level embeddings are often too coarse to capture the specific phonemic distortions typical of dysarthria. Furthermore, many models rely on static alignment tools (such as Kaldi) to map audio to text. These tools frequently fail when processing the irregular timing and prosody of dysarthric speech, leading to poor training signals. DyPCL addresses these limitations by shifting the focus to the phoneme level and introducing dynamic mechanisms for alignment and learning.
Methodology
The DyPCL framework is built upon two "dynamic" pillars designed to improve the precision of phonetic representations:
1. Dynamic CTC Alignment
Rather than using pre-computed, static timestamps from external alignment tools, DyPCL extracts speech embeddings directly from the CTC layer during the training process. This allows the model to refine its alignment as it improves its recognition capabilities. By "learning how to align" in tandem with "learning how to recognize," the framework ensures that the phoneme-to-audio mapping remains accurate even for highly irregular speech patterns.
2. Dynamic Curriculum Learning
The model employs a triplet loss strategy to refine its latent space. In this setup, the model compares an Anchor (healthy speech), a Positive (the same phoneme from a dysarthric speaker), and a Negative (a different phoneme).
To maximize the effectiveness of this learning, DyPCL uses a curriculum-based approach guided by phonetic distance. Using the PanPhon tool to measure the Hamming distance of articulatory features, the training begins with "easy" negatives (phonetically distinct sounds like 'b' vs 's'). As training progresses, the model is challenged with "hard" negatives (phonetically similar sounds like 'æ' vs 'ə'). This forces the model to learn the subtle boundaries between sounds that are most frequently confused in dysarthric speech.
Training is conducted in two stages: an initial standard CTC training phase for basic recognition, followed by a joint CTC and Phoneme-level Contrastive Learning (PCL) phase to sharpen the phonetic clusters.
Experimental Setup
Dataset
The study was benchmarked using the UASpeech dataset, which includes recordings from 13 healthy control speakers and 15 speakers with dysarthria. The dysarthric speakers are categorized by their speech intelligibility levels: High (H), Mid (M), Low (L), and Very Low (VL). The isolated word recordings were decomposed into phonemes using a standard phonemizer for the PCL phase.
Implementation
While DyPCL is compatible with various speech encoders, the authors found that using HuBERT as the backbone provided the strongest performance. The evaluation focused on the model's ability to generalize across different levels of dysarthria severity.
Key Results and Insights
DyPCL demonstrated substantial improvements over baseline models across all metrics:
- Overall Performance: The framework achieved an average 22.10% relative reduction in WER across the entire dataset.
- Impact on Severe Dysarthria: The most significant impact was observed in the "Very Low" intelligibility group, where WER was reduced from 58.49% to 49.45%. This represents a major step forward for the speakers who are typically the most underserved by current ASR technology.
- Phonetic Clustering: UMAP visualizations of the high-dimensional embeddings revealed that DyPCL produces much tighter and more distinct clusters for individual phonemes compared to standard CTC models. This confirms that the model is successfully anchoring dysarthric speech to healthy phonetic "prototypes."
- The Importance of Granularity: The results suggest that phoneme-level analysis is superior to word-level analysis for DSR, as it allows the model to ignore surface-level slurring and focus on the underlying linguistic structure.
Conclusion and Future Work
DyPCL provides a robust framework for addressing the phonetic distortions inherent in dysarthric speech. By combining dynamic alignment with a phonetic curriculum, it bridges the gap between impaired and healthy speech representations.
Future research will focus on several key areas:
- Unpaired Data: Exploring how to apply these techniques to datasets where paired healthy/dysarthric speech is unavailable.
- Data Augmentation: Integrating DyPCL with advanced augmentation strategies to further improve robustness.
- Continuous Speech: Moving beyond the isolated words of the UASpeech dataset to test the framework on the complexities of continuous, conversational speech.
Summary updated