Pre-trained Feature Fusion and Matching for Mild Cognitive Impairment Detection

Junwen Duan, Fangyuan Wei, Hongdong Li, Jin Liu2024 · DOI

Abstract

Effective diagnosis of Mild Cognitive Impairment (MCI), a preclinical stage of cognitive decline, is significant for delaying disease progression. While most current spontaneous speech-based diagnostic methods focus on English speech, the Interspeech 2024 TAUKADIAL Challenge proposed an innovative research direction to develop a language-agnostic approach to diagnose MCI. This paper proposes an MCI diagnosis method by analyzing and combining linguistic and acoustic features using the bilingual Chinese-English speech dataset provided by the challenge. We employed a pre-trained multilingual model and expressivity encoder to extract language-agnostic speech features. To overcome the challenges of data scarcity and language diversity, we implemented data augmentation and alignment to enhance the model's generalization. Our approach achieved 77.5% accuracy, demonstrating its effectiveness and potential on cross-lingual data.

Introduction

Detecting Mild Cognitive Impairment (MCI) early is a critical goal for dementia research, offering a window for interventions that can delay the progression of diseases like Alzheimer’s. While spontaneous speech analysis has emerged as a powerful, non-invasive diagnostic tool, most existing models are optimized for a single language and struggle with the disfluencies inherent in spontaneous speech. This study presents a novel framework that combines multilingual pre-trained models with an "audio-text matching" strategy to create a system capable of operating across both English and Chinese.

Methodology

The proposed framework utilizes a dual-track pipeline to process speech before multimodal fusion.

Feature Extraction and Prompted ASR

To capture the subtle markers of cognitive decline, the researchers employed Whisper large-v3 with a "prompted ASR" technique. Unlike standard ASR which cleans up speech, this method uses specific prompts to ensure the transcription retains disfluencies like "ums," "uhs," and long pauses.

  • Linguistic Track: Uses XLM-RoBERTa-base, a multilingual model, to extract semantic features from the disfluency-rich transcriptions.
  • Acoustic Track: Employs Prosody UnitY2 to capture language-independent prosodic features such as rhythm and speech rate.
The multimodal framework architecture, illustrating the parallel feature extraction and fusion process.

Audio-Text Matching (ATM) Strategy

A core innovation of this work is the Audio-Text Matching (ATM) strategy. To combat data scarcity, the model is trained to determine if a piece of audio matches its transcription using both matched and mismatched pairs. This serves as a powerful data augmentation technique and a regularizer that forces the model to learn deeper cross-modal alignments across different languages.

Illustration of the Audio-Text Matching (ATM) strategy used for data augmentation and alignment across English (en) and Chinese (zh) samples.

Experimental Results

The study utilized the TAUKADIAL Challenge dataset, consisting of 387 training recordings and 120 test recordings balanced for age and gender. The results demonstrated that linguistic features (UAR 0.678) were significantly more predictive than acoustic-only features (UAR 0.554).

Method Accuracy Unweighted Average Recall (UAR) F1 Score
Baseline - 0.592 -
Acoustic-only 0.550 0.554 0.526
Text-only 0.675 0.678 0.667
Multimodal + ATM (Final) 0.775 0.776 0.781

The final multimodal approach with ATM loss achieved 77.5% accuracy, a substantial improvement over the challenge baseline. This success suggests that the structural and disfluency patterns in speech are vital indicators of MCI that can be captured effectively through cross-modal alignment.

Conclusion

This research demonstrates that preserving speech disfluencies and employing cross-modal matching can significantly enhance the accuracy of language-agnostic MCI detection. While currently validated on English and Chinese, future work aims to extend this framework to a wider variety of language families to further improve its global applicability.

Summary updated

Abstract

Effective diagnosis of Mild Cognitive Impairment (MCI), a preclinical stage of cognitive decline, is significant for delaying disease progression.While most current spontaneous speechbased diagnostic methods focus on English speech, the Interspeech 2024 TAUKADIAL Challenge proposed an innovative research direction to develop a language-agnostic approach to diagnose MCI.This paper proposes an MCI diagnosis method by analyzing and combining linguistic and acoustic features using the bilingual Chinese-English speech dataset provided by the challenge.We employed a pre-trained multilingual model and expressivity encoder to extract language-agnostic speech features.To overcome the challenges of data scarcity and language diversity, we implemented data augmentation and alignment to enhance the model's generalization.Our approach achieved 77.5% accuracy, demonstrating its effectiveness and potential on cross-lingual data.
BibTeX
@misc{duan2024pre,
  author = {Junwen Duan and Fangyuan Wei and Hongdong Li and Jin Liu},
  title = {{Pre-trained Feature Fusion and Matching for Mild Cognitive Impairment Detection}},
  year = {2024},
  doi = {10.21437/interspeech.2024-2386},
}
Figure