Machine-learning assisted swallowing assessment: a deep learning-based quality improvement tool to screen for post-stroke dysphagia

Rami Saab, Arjun Balachandar, Hamza Mahdi, Eptehal Nashnoush, Lucas X. Perri, Ashley Waldron, Alireza Sadeghian, Gordon D. Rubenfeld, Mark Crowley, Mark I. Boulos, Brian J. Murray, Houman KhosravaniFrontiers in Neuroscience 2023 · DOI

TL;DR

Researchers have developed a proof-of-concept deep learning tool that screens for post-stroke dysphagia (swallowing difficulty) by analyzing patient voice recordings. By converting speech into Mel-spectrograms and using an ensemble of convolutional neural networks (CNNs), the system achieved a participant-level sensitivity of 89% and an AUC of 0.91, offering a non-invasive way to automate and standardize bedside screening.

What Problem Does This Paper Address?

Dysphagia—the medical term for swallowing dysfunction—affects approximately 55% of acute stroke patients. If left undetected, it leads to aspiration pneumonia, malnutrition, and increased mortality.

The current "gold standard" for assessment is a Video Fluoroscopic Swallowing Study (VFSS), which requires specialized x-ray equipment and personnel. Because VFSS isn't always available, hospitals rely on bedside screening tools like the Toronto Bedside Swallowing Screening Test (TOR-BSST©). However, these screenings are subjective, require specific training/licensure, and can be difficult to administer consistently in high-volume or low-resource settings.

The authors sought to bridge this "quality gap" by using voice as a digital biomarker. Since the neural pathways for speech and swallowing overlap, changes in voice quality (dysphonia) often signal swallowing issues.

Methods

The researchers built a three-step data processing pipeline: Segmentation, Transformation, and Machine Learning.

  1. Data Collection: Using an iPhone 12, they recorded patients performing standardized speech tasks: sustained vowels (/a/, /e/, /i/, /o/, /u/) and sentences/words from the National Institute of Health Stroke Scale (NIHSS).
  2. Transformation: Audio clips were segmented into 0.5-second windows and converted into Mel-spectrograms. These are visual representations of sound that mimic the human ear’s non-linear perception of pitch and frequency.
  3. The Models: The team employed two state-of-the-art CNN architectures—DenseNet-121 and ConvNext-Tiny—using transfer learning (pre-training on the ImageNet dataset).
  4. Ensemble Fusion: They used an "unweighted averaging" ensemble method, combining the outputs of both models to improve robustness and reduce variance.
  5. Voting: To move from "clip-level" predictions (is this 0.5s of sound healthy?) to "participant-level" predictions (does this patient have dysphagia?), they averaged the output probabilities of all clips from a single patient.

Datasets

The study was conducted at Sunnybrook Health Sciences Centre in Toronto.

  • Participants: 70 patients were recruited; 68 were included in the final analysis.
  • Split: 40 patients were used for the training cohort (Epoch 1) and 28 for the testing cohort (Epoch 2).
  • Data Volume: The speech was segmented into 1,579 audio clips, resulting in 6,655 Mel-spectrogram images.
  • Ground Truth: Labels (Pass/Fail) were determined by trained assessors using the validated TOR-BSST© protocol.

Key Insights and Ideas

  • Voice as a Proxy: The study reinforces the idea that vocalizations can serve as a reliable proxy for internal physiological functions like swallowing.
  • Why Mel-spectrograms? Instead of feeding raw audio waveforms into a model, converting them to images allows researchers to leverage powerful computer vision architectures (like CNNs) that are already highly optimized for pattern recognition.
  • Ensemble Advantage: The fusion model outperformed individual models. This suggests that different architectures (DenseNet’s dense connections vs. ConvNext’s transformer-inspired kernels) capture slightly different nuances in the audio data.
  • Speech vs. Vowels: Interestingly, using a combination of vowels and NIHSS speech tasks yielded significantly better results than using vowels alone. This suggests that the complexity of connected speech provides more "signal" for detecting dysphagia than simple phonation.

Key Results

The ensemble fusion model demonstrated strong performance, particularly at the participant level:

  • Participant-Level Performance:
    • Sensitivity: 89% (Crucial for a screening tool to ensure few cases are missed).
  • Specificity: 79%
    • AUC: 0.91 [95% CI: 0.77–1.05]
    • F1 Score: 0.81
  • Clip-Level Performance:
    • Sensitivity: 71%
    • Specificity: 77%
    • AUC: 0.80

The high sensitivity at the participant level suggests this tool is highly effective as a "first-pass" screen to identify patients who need a more formal evaluation by a Speech-Language Pathologist.

Limitations and Future Work

  • Sample Size: With only 68 patients, the study is a "proof-of-concept." Larger, multi-center datasets are needed to ensure the model generalizes across different populations.
  • Manual Segmentation: Currently, the audio clips are segmented manually using software. For a real-world clinical tool, this process would need to be fully automated.
  • Language and Accents: The study focused on primary English speakers. Future iterations must account for different languages, accents, and dental prostheses, which all affect vocal characteristics.
  • Passive Monitoring: The authors suggest that in the future, such tools could be integrated into "passive monitoring" systems in hospital rooms to alert staff to changes in a patient's swallowing status in real-time.

Open Source Note: The authors have made their code available on GitHub to facilitate further research: masa-open-source.

Summary updated

Abstract

This study is the first to demonstrate the feasibility of applying deep learning to classify vocalizations to detect post-stroke dysphagia. Our findings suggest potential for enhancing dysphagia screening in clinical settings. https://github.com/UofTNeurology/masa-open-source.
BibTeX
@inproceedings{saab2023machine,
  author = {Rami Saab and Arjun Balachandar and Hamza Mahdi and Eptehal Nashnoush and Lucas X. Perri and Ashley Waldron and Alireza Sadeghian and Gordon D. Rubenfeld and Mark Crowley and Mark I. Boulos and Brian J. Murray and Houman Khosravani},
  title = {{Machine-learning assisted swallowing assessment: a deep learning-based quality improvement tool to screen for post-stroke dysphagia}},
  booktitle = {Frontiers in Neuroscience},
  year = {2023},
  doi = {10.3389/fnins.2023.1302132},
}
Figure