Improving Dysarthric Speech Segmentation With Emulated and Synthetic Augmentation

Saeid Alavi Naeini, Leif Simmatis, Deniz Jafari, Yana Yunusova, Babak TaatiIEEE Journal of Translational Engineering in Health and Medicine 2024 · DOI

TL;DR

Researchers have developed a way to accurately segment dysarthric speech—speech impaired by neurological conditions—by fine-tuning AI models on "emulated" and synthetic data. By using healthy individuals mimicking speech impairments and AI-generated voices, they achieved a word error rate of just 5.7%, proving that we can build effective clinical tools without needing massive amounts of hard-to-collect patient data.

What Problem Does This Paper Address?

Acoustic analysis is a powerful, non-invasive tool for diagnosing and monitoring neurological diseases like ALS, Parkinson’s, and stroke. However, before a computer can analyze speech features (like pause duration or speaking rate), it first needs to "parse" or segment the audio into individual words.

While modern Automatic Speech Recognition (ASR) models are incredible at transcribing healthy speech, they often fail when faced with dysarthria—the slurred or slow speech patterns common in neurological disorders. To fix this, models usually need to be fine-tuned on real clinical data. The catch? Clinical data is scarce, expensive to collect, and fraught with privacy concerns. This paper asks: Can we skip the clinical data bottleneck by using "fake" dysarthric speech instead?

Methods

The researchers proposed an automated segmentation pipeline consisting of two main stages:

  1. Automatic Speech Recognition (ASR): They used a pre-trained Wav2Vec 2.0 (W2V) model, a state-of-the-art transformer-based framework. They fine-tuned this model on various combinations of real, emulated, and synthetic data to see which training set produced the best results.
  2. Sequence-to-Sequence Alignment: Once the ASR model identifies the words, the system uses Dynamic Time Warping (DTW) to align a "clean" reference recording with the patient's target recording. This allows the system to pinpoint the exact start and end timestamps for every repetition of a phrase.

The specific task used was the repetition of the sentence "Buy Bobby a Puppy" (BBP), a standard phrase in orofacial clinical examinations because of its specific phonetic makeup.

Datasets

The study utilized three distinct data sources:

  • TNF$^x$ (Extended Toronto NeuroFace): The "real-world" benchmark containing recordings from 68 participants, including those with ALS, Parkinson’s, and post-stroke symptoms, as well as healthy controls.
  • App Dataset (Emulated): Recordings from 20 healthy individuals who were asked to intentionally vary their speaking rate (twice as fast, half as slow) and loudness. This "emulates" the variability found in pathological speech.
  • TTS Dataset (Synthetic): 3,663 artificially produced clips using Google Text-to-Speech and Tacotron2. This set included 55 different accents and three speaking rates to maximize vocal diversity.

Key Insights and Ideas

  • Emulation is a High-Quality Signal: One of the most striking findings was that healthy people "acting" out different speaking rates and volumes (the App dataset) provided a training signal almost as effective as real clinical data.
  • Synthetic Diversity Matters: Using TTS data wasn't just about volume; it was about fairness and robustness. By including 55 different accents in the synthetic data, the model became better at handling the diverse vocal representations it might encounter in a global telehealth setting.
  • The "Data Scarcity" Myth: The results suggest that for specific clinical tasks (like segmenting a known phrase), we don't actually need thousands of hours of rare patient recordings. Smart augmentation can bridge the gap, making it much easier to deploy these tools in clinics.

Key Results

The researchers evaluated the models using Word Error Rate (WER), Correct Count Accuracy (CCA), and Intersection over Union (IoU) for temporal alignment.

  • Top Performance: The best-performing model was fine-tuned on a combination of Synthetic (TTS) + Emulated (App) data.
  • Accuracy: This model achieved a Word Error Rate of 5.75% and a Correct Count Accuracy of 94.44% on held-out clinical data.
  • Segmentation Precision: For the actual task of marking word boundaries, the model achieved a mean IoU of 89.62% compared to manual "ground truth" parsing by humans.
  • Clinical Impact: The model performed consistently well across different diseases, though it struggled slightly more with Parkinson’s Disease (PD) due to the "mumbling" and rapid-fire speech patterns (hypokinetic dysarthria) typical of the condition.

Limitations and Future Work

The primary limitation is the scope of the speech task: the model was only trained and tested on a single sentence ("Buy Bobby a Puppy"). While this is a standard clinical tool, real-world applications would eventually need to handle free-form speech or a wider variety of phrases.

Additionally, the study used a single ASR architecture (Wav2Vec 2.0). The authors suggest that future work should explore newer models and larger, more diverse clinical datasets to establish the "optimal" amount of data needed for fine-tuning. They also aim to move toward a more "universally applicable" solution that can categorize dysarthria subtypes automatically.

Summary updated

Abstract

Acoustic features extracted from speech can help with the diagnosis of neurological diseases and monitoring of symptoms over time. Temporal segmentation of audio signals into individual words is an important pre-processing step needed prior to extracting acoustic features. Machine learning techniques could be used to automate speech segmentation via automatic speech recognition (ASR) and sequence to sequence alignment. While state-of-the-art ASR models achieve good performance on healthy speech, their performance significantly drops when evaluated on dysarthric speech. Fine-tuning ASR models on impaired speech can improve performance in dysarthric individuals, but it requires representative clinical data, which is difficult to collect and may raise privacy concerns. This study explores the feasibility of using two augmentation methods to increase ASR performance on dysarthric speech: 1) healthy individuals varying their speaking rate and loudness (as is often used in assessments of pathological speech); 2) synthetic speech with variations in speaking rate and accent (to ensure more diverse vocal representations and fairness). Experimental evaluations showed that fine-tuning a pre-trained ASR model with data from these two sources outperformed a model fine-tuned only on real clinical data and matched the performance of a model fine-tuned on the combination of real clinical data and synthetic speech. When evaluated on held-out acoustic data from 24 individuals with various neurological diseases, the best performing model achieved an average word error rate of 5.7% and a mean correct count accuracy of 94.4%. In segmenting the data into individual words, a mean intersection-over-union of 89.2% was obtained against manual parsing (ground truth). It can be concluded that emulated and synthetic augmentations can significantly reduce the need for real clinical data of dysarthric speech when fine-tuning ASR models and, in turn, for speech segmentation.
BibTeX
@article{naeini2024improving,
  author = {Saeid Alavi Naeini and Leif Simmatis and Deniz Jafari and Yana Yunusova and Babak Taati},
  title = {{Improving Dysarthric Speech Segmentation With Emulated and Synthetic Augmentation}},
  journal = {IEEE Journal of Translational Engineering in Health and Medicine},
  year = {2024},
  doi = {10.1109/jtehm.2024.3375323},
}
Figure