Interpretable Speech Features vs. DNN Embeddings: What to Use in the Automatic Assessment of Parkinson’s Disease in Multi-lingual Scenarios

Anna Favaro, Yi-Ting Tsai, Ankur Butala, Thomas Thebaud, Jesús Villalba, Najim Dehak, Laureano Moro-Velázquez

Abstract

Individuals with Parkinson’s disease (PD) develop speech impairments that deteriorate their communication capabilities. Speech-based approaches for PD assessment rely on feature extraction for automatic classification or detection. It is desirable for these features to be interpretable to facilitate their development as diagnostic tools in clinical environments. However, many studies propose detection techniques based on non-interpretable embeddings from Deep Neural Networks since these provide high detection accuracy, and do not compare them with the performance of interpretable features for the same task. The goal of this work was twofold: providing a systematic comparison between the predictive capabilities of models based on interpretable and non-interpretable features and exploring the language robustness of the features themselves. As interpretable features, prosodic, linguistic, and cognitive descriptors were employed. As non-interpretable features, x-vectors, Wav2Vec 2.0, HuBERT, and TRILLsson representations were used. To the best of our knowledge, this is the first study applying TRILLsson and HuBERT to PD detection. Mono-lingual, multi-lingual, and cross-lingual machine learning experiments were conducted on six data sets. These contain speech recordings from different languages: American English, Castilian Spanish, Colombian Spanish, Italian, German, and Czech. For interpretable feature-based models, the mean of the best F1-scores obtained from each language was 81% in mono-lingual, 81% in multi-lingual, and 71% in cross-lingual experiments. For non-interpretable feature-based models, instead, they were 85% in mono-lingual, 88% in multi-lingual, and 79% in cross-lingual experiments. On one hand, models based on non-interpretable features outperformed interpretable ones, especially in cross-lingual experiments. Among the non-interpretable features used, TRILLsson provided the most stable and accurate results across tasks and data sets. Conversely, the two types of features adopted showed some level of language robustness in multi-lingual and cross-lingual experiments. Overall, these results suggest that interpretable feature-based models can be used by clinicians to evaluate the evolution and the possible deterioration of the speech of patients with PD, while non-interpretable feature-based models can be leveraged to achieve higher detection accuracy.

Introduction

Parkinson’s Disease (PD) often manifests early as voice and speech disorders, collectively known as hypokinetic dysarthria. While clinicians use subjective rating scales to track progression, these can lack sensitivity at early stages. Automated speech-based assessment offers a non-invasive, objective alternative. However, the field is currently split between two methodologies: Interpretable Feature-based Models (IFMs) and Non-Interpretable Feature-based Models (NIFMs). IFMs utilize hand-crafted metrics like pitch jitter or pause frequency that clinicians can understand, whereas NIFMs leverage high-dimensional embeddings from deep learning that often boast superior accuracy but lack transparency.

Methodology

The researchers compared these two philosophies across three experimental scenarios:

  • Mono-lingual: Models trained and tested on a single language.
  • Multi-lingual: Models trained on all available languages and tested on each individually.
  • Cross-lingual: Models trained on all but one language and tested on the unseen language.

Feature Extraction

  • Interpretable Features: 125 prosodic features (pitch, loudness, pauses), 13 linguistic features (part-of-speech tags, syntactic complexity), and 4 cognitive features (speech rhythm and "informational units" from picture descriptions).
  • Non-Interpretable Features: State-of-the-art self-supervised learning (SSL) representations, including x-vectors, Wav2Vec 2.0, HuBERT, and TRILLsson. This study marks the first application of TRILLsson and HuBERT to PD detection.

Data Corpora

The study utilized six distinct corpora covering a diverse linguistic landscape: American English (NLS), Castilian Spanish (Neurovoz), Colombian Spanish (GITA), German (GermanPD), Italian (ItalianPVS), and Czech (CzechPD). The recordings included three types of tasks: spontaneous speech, reading passages, and text-dependent utterances.

Distribution of speech tasks across the six datasets used in the study.

Results and Discussion

The results highlight a clear accuracy-interpretability trade-off. NIFMs consistently outperformed IFMs across all scenarios:

  • Mono-lingual: NIFMs achieved a mean F1-score of 85% vs. 81% for IFMs.
  • Multi-lingual: The gap widened, with NIFMs reaching 88% F1-score compared to 81% for IFMs.
  • Cross-lingual: NIFMs remained more robust, achieving a 79% F1-score, while IFMs dropped to 71%.

Among the "black box" models, TRILLsson—a model specifically distilled for paralinguistic tasks—was the standout performer. Its stability and accuracy suggest that PD markers are deeply embedded in non-lexical, paralinguistic aspects of speech. Furthermore, training on multiple languages simultaneously improved performance for both model types, suggesting that PD-related speech impairments share "universal" patterns that transcend specific language boundaries.

Conclusion

While NIFMs are superior "detectors" for high-accuracy screening, IFMs remain essential "diagnosticians." IFMs allow clinicians to monitor specific symptoms, such as changes in noun-to-verb ratios or pause durations, which is vital for tailoring treatment. Future work should address real-world complexities like medication effects and the challenge of differential diagnosis against "mimic" conditions like Essential Tremor.

Summary updated

Abstract

Abstract Individuals with Parkinson’s disease (PD) develop speech impairments that deteriorate their communication capabilities. Speech-based approaches for PD assessment rely on feature extraction for automatic classification or detection. It is desirable for these features to be interpretable to facilitate their development as diagnostic tools in clinical environments. However, many studies propose detection techniques based on non-interpretable embeddings from Deep Neural Networks since these provide high detection accuracy, and do not compare them with the performance of interpretable features for the same task. The goal of this work was twofold: providing a systematic comparison between the predictive capabilities of models based on interpretable and non-interpretable features and exploring the language robustness of the features themselves. As interpretable features, prosodic, linguistic, and cognitive descriptors were employed. As non-interpretable features, x-vectors, Wav2Vec 2.0, HuBERT, and TRILLsson representations were used. To the best of our knowledge, this is the first study applying TRILLsson and HuBERT to PD detection. Mono-lingual, multi-lingual, and cross-lingual machine learning experiments were conducted on six data sets. These contain speech recordings from different languages: American English, Castilian Spanish, Colombian Spanish, Italian, German, and Czech. For interpretable feature-based models, the mean of the best F1-scores obtained from each language was 81% in mono-lingual, 81% in multi-lingual, and 71% in cross-lingual experiments. For non-interpretable feature-based models, instead, they were 85% in mono-lingual, 88% in multi-lingual, and 79% in cross-lingual experiments. On one hand, models based on non-interpretable features outperformed interpretable ones, especially in cross-lingual experiments. Among the non-interpretable features used, TRILLsson provided the most stable and accurate results across tasks and data sets. Conversely, the two types of features adopted showed some level of language robustness in multi-lingual and cross-lingual experiments. Overall, these results suggest that interpretable feature-based models can be used by clinicians to evaluate the evolution and the possible deterioration of the speech of patients with PD, while non-interpretable feature-based models can be leveraged to achieve higher detection accuracy. Highlights Both interpretable and non-interpretable features displayed robust behaviors. Models based on non-interpretable features outperformed interpretable ones. Interpretable feature-based models provide insights into speech and language deterioration. Non-interpretable feature-based models can be used to achieve higher detection accuracy.
BibTeX
@article{favaro2023interpretable,
  author = {Anna Favaro and Yi-Ting Tsai and Ankur Butala and Thomas Thebaud and Jesús Villalba and Najim Dehak and Laureano Moro-Velázquez},
  title = {{Interpretable Speech Features vs. DNN Embeddings: What to Use in the Automatic Assessment of Parkinson’s Disease in Multi-lingual Scenarios}},
  journal = {Computers in Biology and Medicine},
  year = {2023},
  url = {https://openreview.net/forum?id=veWJqTXCE3},
}
Figure