Improving End-to-End Speech Recognition for Dysarthric Speech through In-Domain Data Augmentation

Paban Sapkota, Hemant Kumar Kathania, Sudarsana Reddy Kadiri, Shrikanth Narayanan

TL;DR

This paper tackles the challenge of accurately recognizing dysarthric speech, which is often difficult to understand due to motor impairments. The researchers demonstrate that augmenting existing speech data with techniques like altering speaking rate and pitch significantly improves the performance of end-to-end speech recognition models, especially when tailored to different severity levels of dysarthria.

What Problem Does This Paper Address?

Dysarthria is a motor speech disorder that affects muscle control, leading to speech that can be slurred, slow, or generally hard to understand. This significantly impacts communication for individuals with the condition. While automatic speech recognition (ASR) has made great strides for typical speech, recognizing dysarthric speech remains a major hurdle. This is largely due to the inherent variability in dysarthric speech, which can differ greatly in severity and acoustic properties, and the scarcity of labeled data for training robust ASR models. Existing powerful pre-trained models like Wav2Vec2 show promise, but effectively fine-tuning them for dysarthric speech, especially across different severity levels, requires overcoming data limitations.

Methods

The core of this research lies in exploring data augmentation techniques to enhance the fine-tuning of pre-trained end-to-end ASR models, specifically Wav2Vec2, for dysarthric speech. The authors investigate four distinct augmentation methods, each targeting a different aspect of speech production that can be affected by dysarthria:

  • Speaking-Rate Modification (SRM): Adjusts the speed of speech using Time Scale Modification (TSM).
  • Pitch Modification (PM): Alters the fundamental frequency (pitch) of the speech.
  • Formant Modification (FM): Modifies the spectral characteristics of speech by adjusting formant frequencies, which are crucial for vowel intelligibility. This is achieved using Linear Predictive Coding (LPC) models.
  • Vocal Tract Length Perturbation (VTLP): Simulates variations in vocal tract length, which can affect spectral envelopes and mimic inter-speaker variability.

These methods were applied to the training data, while the test data remained untouched to ensure fair evaluation. The researchers also performed a severity-based analysis, training models on data from one severity level and evaluating them on others to assess generalization capabilities. They systematically determined the optimal modification factor for each augmentation technique for each severity level.

Datasets

The study utilized the TORGO database, a widely used corpus of dysarthric speech. This database contains approximately 15 hours of data from 15 speakers, including 8 dysarthric speakers and 7 age-matched healthy controls. The dysarthric speakers' severity levels were assessed using the Franchey Dysarthria Assessment (FDA) and categorized into low, medium, and high severity.

Key Insights and Ideas

  • Severity Matters: The effectiveness of different augmentation techniques varies significantly depending on the severity level of the dysarthric speech. What works best for low-severity speech might not be optimal for high-severity speech.
  • Targeted Augmentation: Each augmentation method targets a specific acoustic characteristic affected by dysarthria. SRM and PM seem to be particularly effective for addressing the overall intelligibility issues, while FM and VTLP might help in refining specific phonetic aspects.
  • Data Augmentation as a Solution: The research strongly suggests that data augmentation is a viable and effective strategy to combat data scarcity and improve the robustness of ASR systems for dysarthric speech, especially when using powerful pre-trained models.
  • Cross-Severity Generalization: Training on augmented data, particularly from higher severity levels, can lead to improved performance even on lower severity test sets, hinting at potential for broader generalization.

Key Results

The study found that data augmentation significantly improved ASR performance compared to baseline models fine-tuned without augmentation.

  • Best Overall Performance:
    • For low severity speech, SRM with a modification factor of s=0.8 achieved the lowest Word Error Rate (WER) of 9.02%, representing a relative improvement of 30.02%.
  • For medium severity speech, SRM with s=0.8 also performed best, achieving a WER of 38.11%, a relative improvement of 16.64%.
  • For high severity speech, PM with a modification factor of τ=0.8 yielded the best results with a WER of 55.15%, a relative improvement of 15.47%.

The paper also highlights that fine-tuning with augmented high-severity data improved performance on low and medium severity test sets, suggesting a degree of generalization.

Limitations and Future Work

  • Severity-Specific Models: The current study relies on known severity labels for severity-specific fine-tuning. In real-world applications, automatically classifying speaker severity would be crucial.
  • Future Directions: The authors suggest future research should focus on developing automatic severity classification methods using SSL features and integrating this into end-to-end ASR systems. They also mention that their other work has explored acoustic models and features in traditional Kaldi-based ASR systems, which could be a complementary research avenue.

Summary updated

Abstract

Dysarthric speech recognition is crucial for facilitating effective communication among individuals with dysarthria. However, accurately recognizing dysarthric speech poses significant challenges due to varying severity levels and limited data availability. In this paper, we explore data augmentation techniques for dysarthric automatic speech recognition (ASR) systems by fine-tuning the End-to-End pre-trained Wav2Vec2 model, with a specific focus on severity levels. To address the challenges of data scarcity and the need for extensive data in fine-tuning pre-trained ASR systems for dysarthric speech, we investigate four prominent data augmentation methods: Speaking-Rate Modification (SRM), Pitch Modification (PM), Formant Modification (FM), and vocal tract Length Perturbation (VTLP), tailored to different aspects of dysarthria. The study uses individually fine-tuned Wav2Vec2 models for each severity class as baseline systems. Additionally, we conducted severity-specific fine-tuning of the ASR model using augmented data. Results demonstrate distinct efficacy patterns for each augmentation technique across severity levels. The best WERs were achieved with SRM (s=0.8) low (9.02%) and medium (38.11 %) severities, and with PM (τ=0.8) for high severity (55.15 %), reflecting relative improvements of 30.02%, 16.64%, and 15.47%, respectively. These results confirm the effectiveness of the augmentation methods in improving dysarthric ASR performance.
BibTeX
@inproceedings{sapkota2024improving,
  author = {Paban Sapkota and Hemant Kumar Kathania and Sudarsana Reddy Kadiri and Shrikanth Narayanan},
  title = {{Improving End-to-End Speech Recognition for Dysarthric Speech through In-Domain Data Augmentation}},
  booktitle = {IEEECONF},
  year = {2024},
  url = {https://openreview.net/forum?id=rA2wLsCzJw},
}
Figure