Comparison of Speech Tasks in Human Expert and Machine Detection of Parkinson's Disease

Peter Plantinga, Roozbeh Sattari, Karine Marcotte, Carla Di Gironimo, Madeleine Sharp, Liziane Bouvier, Maiya Geddes, Ingrid Verduyckt, Étienne de Villers-Sidani, Mirco Ravanelli, Denise KleinarXiv 2025 · DOI · arXiv

TL;DR

This paper compares how well human experts (neurologists and speech-language pathologists) and a machine learning model (Whisper) can detect Parkinson's Disease (PD) from speech samples. The study reveals that while humans leverage broader contextual information, Whisper performs comparably or even better than experts when relying solely on audio, particularly in challenging cases like mild PD, younger patients, and spontaneous speech.

What Problem Does This Paper Address?

Diagnosing Parkinson's Disease (PD) is a significant challenge, with even experienced clinicians exhibiting imperfect accuracy. This can lead to diagnostic delays and increased healthcare costs. Speech abnormalities are known indicators of PD, offering a potential low-cost avenue for early detection and monitoring. While machine learning (ML) models, especially foundation models like Whisper, show promise in identifying these speech biomarkers, the specific acoustic features they rely on and how these compare to human expert reasoning remain poorly understood. This paper aims to bridge this gap by directly comparing human expert and ML model performance on PD detection from speech, focusing on the acoustic dimension of clinical judgment.

Methods

The researchers employed a two-pronged approach:

  1. Human Expert Listening Tests: Seven experienced clinicians (4 speech-language pathologists, 3 neurologists) listened to 64 audio samples (30 seconds or less) from individuals with and without PD. For each sample, they made a binary prediction (PD or healthy control), rated their confidence, and provided a reason for their decision (e.g., voice quality, speech prosody, language use).
  2. Machine Learning Model: A small classification module was trained on top of a frozen Whisper Small encoder. This approach leverages Whisper's pre-trained capabilities while minimizing the number of trainable parameters, aiming to preserve the model's inherent understanding of speech. The model was trained on a subset of the Quebec Parkinson Network (QPN) dataset and tested on samples that were also reviewed by human experts.

Datasets

The study utilized the Quebec Parkinson Network (QPN) dataset. This dataset comprises speech recordings from 208 patients with Parkinson's Disease and 52 healthy controls. The recordings were made in a quiet room using a headset microphone, with most PD patients in the "ON" medication state. Participants performed five distinct speech tasks: sustained vowel phonation (SVP), sentence repetition, reading a short passage, recalling a memory, and describing a picture (DPT).

Key Insights and Ideas

  • Whisper's Acoustic Prowess: When restricted to audio-only analysis, Whisper demonstrates performance comparable to, and in some cases superior to, human experts. This highlights the potential of ML models to extract subtle acoustic cues that might be missed or less emphasized by humans.
  • Complementary Strengths: The paper emphasizes that human experts integrate multimodal cues and contextual knowledge (patient history, visual information) that go beyond speech alone. ML models, in this context, excel at isolating and analyzing acoustic irregularities.
  • Challenging Cases: Whisper shows a particular advantage in detecting PD in younger patients, mild cases, and female patients, as well as in spontaneous speech tasks. These are often areas where human diagnosis can be more challenging.
  • Language Nuances: Human experts reported more difficulty identifying PD in individuals speaking a second language, as distinguishing between PD-related speech changes and language interference was challenging. Whisper effectively closed this gap, suggesting it is less susceptible to these linguistic confounds.
  • Explainability Gap: A significant limitation noted is the "black box" nature of ML models. While clinicians can articulate their reasoning, understanding why a model makes a specific prediction is crucial for practical application and trust. This paper is presented as an early step towards addressing this.

Key Results

  • Overall Performance: Whisper and human experts performed similarly across most tested categories.
  • Task-Specific Performance: Whisper outperformed human experts on "Recall" and "Picture Description" tasks, which are spontaneous speech tasks.
  • Demographic Performance: Whisper showed better performance than human experts in the youngest age group (53-62 years old) and for "mild" PD cases.
  • Sex and Language: While both human experts and Whisper performed robustly across sexes, Whisper demonstrated a notable advantage in identifying PD in individuals speaking a second language (L2+), where human expert performance was lower.
  • Reasoning Breakdown: Human experts primarily cited "Voice Quality" (95% for SVP, 64% for Recall, 56% for DPT) as their decision reason. Prosody was also a factor, particularly in spontaneous speech tasks. Whisper's performance was slightly better on samples where prosody was a key factor, suggesting it might leverage prosodic cues more effectively.

Limitations and Future Work

  • Artificial Scenario: The study acknowledges that the comparison is based on an artificial scenario where both humans and machines are restricted to audio-only analysis. In real-world clinical practice, humans utilize a wealth of multimodal and contextual information.
  • Explainability: The "black box" nature of ML models is a significant limitation. Future work is needed to develop methods for interpreting model predictions and aligning them with clinical reasoning.
  • Accessibility: While Whisper shows promise for improving accessibility by analyzing speech alone, further research is required to understand how these models can be practically integrated into clinical workflows to assist clinicians.
  • Data Augmentation: The paper mentions data augmentation techniques (adding noise, dropping frequencies, dropping audio chunks) used to prevent overfitting, which is a standard practice but worth noting for reproducibility.

Summary updated

BibTeX
@misc{plantinga2025comparison,
  author = {Peter Plantinga and Roozbeh Sattari and Karine Marcotte and Carla Di Gironimo and Madeleine Sharp and Liziane Bouvier and Maiya Geddes and Ingrid Verduyckt and Étienne de Villers-Sidani and Mirco Ravanelli and Denise Klein},
  title = {{Comparison of Speech Tasks in Human Expert and Machine Detection of Parkinson's Disease}},
  year = {2025},
  eprint = {2510.07299},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.07299},
}
Figure