Introduction
Pathological speech analysis—the use of vocal biomarkers to diagnose medical conditions—has traditionally relied on a "one model, one disease" paradigm. Researchers typically develop bespoke architectures for specific conditions like Alzheimer’s Disease (AD) or Major Depressive Disorder (MDD). This approach is inherently inefficient, requiring separate large-scale models for every condition, and it fails to leverage the common acoustic markers—such as changes in pitch, rhythm, or fluency—shared across various speech-affecting pathologies.
To address these limitations, this work introduces a unified framework capable of detecting multiple diseases across different languages using a single model. By employing prompt tuning on a pre-trained spoken language model, the system achieves state-of-the-art performance while modifying only a tiny fraction of the model's parameters.
Methodology
The proposed framework is built upon the Generative Spoken Language Model (GSLM) and utilizes a three-stage pipeline:
- Speech-to-Unit Module: Audio signals are converted into discrete "speech units" using HuBERT (a self-supervised speech representation model) and k-means clustering. This discretization allows the speech to be processed similarly to text tokens in a standard language model.
- Unit-Language Model (uLM) with Prompt Tuning: The core of the system is a 12-layer Transformer (uLM) that remains frozen. Adaptation to specific tasks is achieved by training a small set of prefix prompts. These prompts are composed of three distinct embeddings: disease-specific, language, and class-specific prompts.
- Verbalizer and Voting Mechanism: A fully connected layer (verbalizer) maps internal tokens to disease labels. Given the length of medical recordings, the system segments audio, generates predictions for each segment, and employs a voting mechanism to determine the final speaker-level diagnosis.
Datasets and Evaluation
The model was evaluated on three diverse datasets:
- MDD Corpus: 1,000 hours of English interviews for Major Depressive Disorder.
- AD Corpus: 400 hours of Mandarin clinical interviews for Alzheimer’s Disease.
- PD Dataset: 12.6 hours of Mandarin recordings for Parkinson’s Disease, including both medicated and unmedicated subjects.
Results and Discussion
The unified model significantly outperformed a standard Convolutional Recurrent Neural Network (CRNN) baseline across all tested conditions.
| Dataset | Metric | Our Model | Baseline (CRNN) |
|---|---|---|---|
| MDD (Depression) | F1 Score | 0.78 | 0.62 |
| AD (Alzheimer's) | F1 Score | 0.70 | 0.39 |
| PD (Parkinson's) | F1 Score | 0.63 | 0.32 |
Key findings include:
- Knowledge Transfer: The unified model's superior performance suggests it learns a generalized representation of "pathological speech" that benefits multiple tasks simultaneously.
- Efficiency of Prompt Tuning: Tuning only the prompts led to faster convergence and higher F1 scores compared to full fine-tuning, which is vital in data-scarce medical domains.
- Impact of Voting: Aggregating segment-level predictions via voting was crucial; for instance, AD detection accuracy improved from 0.58 to 0.73 with this mechanism.
Limitations and Future Work
A notable challenge was observed with the Parkinson’s Disease dataset, where the model showed a high false-positive rate for medicated patients. These patients often retain subtle pathological speech features that the model detects as abnormal, despite being labeled as "negative" (medicated). Future research will focus on distinguishing between medicated and healthy speech and expanding the framework to a wider array of speech disorders and languages.
Summary updated