Abstract
Effective diagnosis of Mild Cognitive Impairment (MCI), a preclinical stage of cognitive decline, is significant for delaying disease progression. While most current spontaneous speech-based diagnostic methods focus on English speech, the Interspeech 2024 TAUKADIAL Challenge proposed an innovative research direction to develop a language-agnostic approach to diagnose MCI. This paper proposes an MCI diagnosis method by analyzing and combining linguistic and acoustic features using the bilingual Chinese-English speech dataset provided by the challenge. We employed a pre-trained multilingual model and expressivity encoder to extract language-agnostic speech features. To overcome the challenges of data scarcity and language diversity, we implemented data augmentation and alignment to enhance the model's generalization. Our approach achieved 77.5% accuracy, demonstrating its effectiveness and potential on cross-lingual data.
Introduction
Detecting Mild Cognitive Impairment (MCI) early is a critical goal for dementia research, offering a window for interventions that can delay the progression of diseases like Alzheimer’s. While spontaneous speech analysis has emerged as a powerful, non-invasive diagnostic tool, most existing models are optimized for a single language and struggle with the disfluencies inherent in spontaneous speech. This study presents a novel framework that combines multilingual pre-trained models with an "audio-text matching" strategy to create a system capable of operating across both English and Chinese.
Methodology
The proposed framework utilizes a dual-track pipeline to process speech before multimodal fusion.
Feature Extraction and Prompted ASR
To capture the subtle markers of cognitive decline, the researchers employed Whisper large-v3 with a "prompted ASR" technique. Unlike standard ASR which cleans up speech, this method uses specific prompts to ensure the transcription retains disfluencies like "ums," "uhs," and long pauses.
- Linguistic Track: Uses XLM-RoBERTa-base, a multilingual model, to extract semantic features from the disfluency-rich transcriptions.
- Acoustic Track: Employs Prosody UnitY2 to capture language-independent prosodic features such as rhythm and speech rate.
Audio-Text Matching (ATM) Strategy
A core innovation of this work is the Audio-Text Matching (ATM) strategy. To combat data scarcity, the model is trained to determine if a piece of audio matches its transcription using both matched and mismatched pairs. This serves as a powerful data augmentation technique and a regularizer that forces the model to learn deeper cross-modal alignments across different languages.
Experimental Results
The study utilized the TAUKADIAL Challenge dataset, consisting of 387 training recordings and 120 test recordings balanced for age and gender. The results demonstrated that linguistic features (UAR 0.678) were significantly more predictive than acoustic-only features (UAR 0.554).
| Method | Accuracy | Unweighted Average Recall (UAR) | F1 Score |
|---|---|---|---|
| Baseline | - | 0.592 | - |
| Acoustic-only | 0.550 | 0.554 | 0.526 |
| Text-only | 0.675 | 0.678 | 0.667 |
| Multimodal + ATM (Final) | 0.775 | 0.776 | 0.781 |
The final multimodal approach with ATM loss achieved 77.5% accuracy, a substantial improvement over the challenge baseline. This success suggests that the structural and disfluency patterns in speech are vital indicators of MCI that can be captured effectively through cross-modal alignment.
Conclusion
This research demonstrates that preserving speech disfluencies and employing cross-modal matching can significantly enhance the accuracy of language-agnostic MCI detection. While currently validated on English and Chinese, future work aims to extend this framework to a wider variety of language families to further improve its global applicability.
Summary updated