Unified Pathological Speech Analysis with Prompt Tuning

Fei Yang, Xuenan Xu, Mengyue Wu, Kai YuarXiv 2024 · DOI · arXiv

Introduction

Pathological speech analysis—the use of vocal biomarkers to diagnose medical conditions—has traditionally relied on a "one model, one disease" paradigm. Researchers typically develop bespoke architectures for specific conditions like Alzheimer’s Disease (AD) or Major Depressive Disorder (MDD). This approach is inherently inefficient, requiring separate large-scale models for every condition, and it fails to leverage the common acoustic markers—such as changes in pitch, rhythm, or fluency—shared across various speech-affecting pathologies.

Unified framework overview for multi-disease detection from speech.

To address these limitations, this work introduces a unified framework capable of detecting multiple diseases across different languages using a single model. By employing prompt tuning on a pre-trained spoken language model, the system achieves state-of-the-art performance while modifying only a tiny fraction of the model's parameters.

Methodology

The proposed framework is built upon the Generative Spoken Language Model (GSLM) and utilizes a three-stage pipeline:

  1. Speech-to-Unit Module: Audio signals are converted into discrete "speech units" using HuBERT (a self-supervised speech representation model) and k-means clustering. This discretization allows the speech to be processed similarly to text tokens in a standard language model.
  2. Unit-Language Model (uLM) with Prompt Tuning: The core of the system is a 12-layer Transformer (uLM) that remains frozen. Adaptation to specific tasks is achieved by training a small set of prefix prompts. These prompts are composed of three distinct embeddings: disease-specific, language, and class-specific prompts.
  3. Verbalizer and Voting Mechanism: A fully connected layer (verbalizer) maps internal tokens to disease labels. Given the length of medical recordings, the system segments audio, generates predictions for each segment, and employs a voting mechanism to determine the final speaker-level diagnosis.
Detailed system architecture including the Speech-to-Unit module, prompt-tuned LM, and voting mechanism.

Datasets and Evaluation

The model was evaluated on three diverse datasets:

  • MDD Corpus: 1,000 hours of English interviews for Major Depressive Disorder.
  • AD Corpus: 400 hours of Mandarin clinical interviews for Alzheimer’s Disease.
  • PD Dataset: 12.6 hours of Mandarin recordings for Parkinson’s Disease, including both medicated and unmedicated subjects.

Results and Discussion

The unified model significantly outperformed a standard Convolutional Recurrent Neural Network (CRNN) baseline across all tested conditions.

Dataset Metric Our Model Baseline (CRNN)
MDD (Depression) F1 Score 0.78 0.62
AD (Alzheimer's) F1 Score 0.70 0.39
PD (Parkinson's) F1 Score 0.63 0.32

Key findings include:

  • Knowledge Transfer: The unified model's superior performance suggests it learns a generalized representation of "pathological speech" that benefits multiple tasks simultaneously.
  • Efficiency of Prompt Tuning: Tuning only the prompts led to faster convergence and higher F1 scores compared to full fine-tuning, which is vital in data-scarce medical domains.
  • Impact of Voting: Aggregating segment-level predictions via voting was crucial; for instance, AD detection accuracy improved from 0.58 to 0.73 with this mechanism.

Limitations and Future Work

A notable challenge was observed with the Parkinson’s Disease dataset, where the model showed a high false-positive rate for medicated patients. These patients often retain subtle pathological speech features that the model detects as abnormal, despite being labeled as "negative" (medicated). Future research will focus on distinguishing between medicated and healthy speech and expanding the framework to a wider array of speech disorders and languages.

Summary updated

Abstract

Pathological speech analysis has been of interest in the detection of certain diseases like depression and Alzheimer's disease and attracts much interest from researchers. However, previous pathological speech analysis models are commonly designed for a specific disease while overlooking the connection between diseases, which may constrain performance and lower training efficiency. Instead of fine-tuning deep models for different tasks, prompt tuning is a much more efficient training paradigm. We thus propose a unified pathological speech analysis system for as many as three diseases with the prompt tuning technique. This system uses prompt tuning to adjust only a small part of the parameters to detect different diseases from speeches of possible patients. Our system leverages a pre-trained spoken language model and demonstrates strong performance across multiple disorders while only fine-tuning a fraction of the parameters. This efficient training approach leads to faster convergence and improved F1 scores by allowing knowledge to be shared across tasks. Our experiments on Alzheimer's disease, Depression, and Parkinson's disease show competitive results, highlighting the effectiveness of our method in pathological speech analysis.
BibTeX
@misc{yang2024unified,
  author = {Fei Yang and Xuenan Xu and Mengyue Wu and Kai Yu},
  title = {{Unified Pathological Speech Analysis with Prompt Tuning}},
  year = {2024},
  eprint = {2411.04142},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.04142},
}
Figure