Uncovering the Potential for a Weakly Supervised End-to-End Model in Recognising Speech from Patient with Post-Stroke Aphasia

Giulia Sanguedolce, Patrick A. Naylor, Fatemeh GeranmayehACL 2023

TL;DR

This paper investigates the effectiveness of OpenAI's Whisper model for transcribing speech from individuals with post-stroke aphasia, a challenging task due to speech variability. They found that while Whisper performs reasonably well, its accuracy is significantly impacted by aphasia severity and lesion location, suggesting avenues for future model improvements tailored to this population.

What Problem Does This Paper Address?

Post-stroke aphasia, a language impairment affecting speech and comprehension, significantly impacts quality of life. While speech and language therapy is crucial, intensive treatment is often inaccessible due to cost and limited resources. Automatic Speech Recognition (ASR) holds promise for improving diagnosis and providing feedback for therapy, but existing ASR models struggle with the highly variable and error-prone speech patterns of aphasic individuals. This variability stems from semantic and phonological errors, dysfluencies, and even motor control issues like apraxia and dysarthria, making it difficult for standard ASR systems to achieve high accuracy. Furthermore, there's a scarcity of large, annotated datasets of aphasic speech for training robust models.

Methods

The researchers evaluated OpenAI's Whisper, a state-of-the-art end-to-end ASR model known for its strong performance on healthy speech, using a novel dataset of post-stroke aphasic speech. They fine-tuned Whisper's hyperparameters, specifically focusing on model size, compression_ratio_threshold, and logprob_threshold, to optimize for the lowest Word Error Rate (WER) on this specific patient population. They then conducted correlation analyses and used linear mixed-effects models to explore the relationships between WER and various patient characteristics, including aphasia severity (measured by the Comprehensive Aphasia Test - CAT), stroke lesion location and size, and demographic factors.

Datasets

The primary dataset used in this study is the SONIVA (Speech recOgNition Validation in Aphasia) database, which the authors are actively building. SONIVA is composed of speech recordings from patients participating in the IC32 and PLORAS studies. For this specific study, they used audio data from 46 participants: 23 individuals with post-stroke aphasia (PWA) and 23 age-matched controls. The speech was collected during a picture description task from the CAT assessment and was transcribed verbatim by trained speech therapists.

Key Insights and Ideas

  • Whisper's Potential, But Not a Panacea: While Whisper shows promise for aphasic speech, it's not a perfect solution out-of-the-box. Its performance is significantly worse than on healthy speech, highlighting the unique challenges posed by aphasia.
  • Aphasia Severity Matters: The more severe a patient's aphasia (as indicated by CAT scores), the higher the Word Error Rate from Whisper. This suggests that ASR systems need to be sensitive to the degree of language impairment.
  • Location, Location, Location: The location of the stroke lesion is a critical factor. ASR performance is notably worse for patients with left-hemisphere lesions, particularly in the left frontal lobe, which aligns with the known localization of expressive language processing in the brain.
  • Beyond Basic WER: The study hints that simply measuring WER might not capture the full picture. Qualitative observations noted Whisper's struggle with false starts and unintelligible words, suggesting that other metrics like confidence scores and Phoneme Error Rate (PER) could be valuable for future evaluations.

Key Results

  • The Word Error Rate (WER) for patients with aphasia was significantly higher (38.5%) compared to age-matched controls (10.3%), representing an almost four-fold increase.
  • All three CAT scores (Total Picture description, Naming, and Comprehension) showed a significant negative correlation with WER, indicating that poorer performance on these language assessments corresponded to higher ASR errors.
  • Stroke lesion volume did not show a significant correlation with ASR performance.
  • Linear mixed-effects models revealed that patients with left temporal and frontal lobe lesions had worse ASR performance.

Limitations and Future Work

The authors acknowledge that the primary limitation of this study is the relatively small size of the speech database used. They are actively working on building a larger, detailed, and annotated speech and language database for patients with post-stroke aphasia. Future research should focus on verifying these findings with larger datasets and assessing the generalizability of Whisper's performance to aphasia resulting from other conditions. Additionally, exploring other ASR metrics like confidence scores and Phoneme Error Rate, and investigating how stroke lesion location can be leveraged to improve ASR training and performance for PWA, are highlighted as important future directions.

Summary updated

BibTeX
@inproceedings{sanguedolce2023uncovering,
  author = {Giulia Sanguedolce and Patrick A. Naylor and Fatemeh Geranmayeh},
  title = {{Uncovering the Potential for a Weakly Supervised End-to-End Model in Recognising Speech from Patient with Post-Stroke Aphasia}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure