TL;DR
This paper introduces an attention-based speech recognition system for Mandarin speakers to assess dementia. By leveraging transfer learning from large speech datasets and fine-tuning on a local dataset of picture description tasks, the system achieves high accuracy in detecting Alzheimer's disease and predicting dementia severity.
What Problem Does This Paper Address?
Diagnosing dementia is a complex and time-consuming process, often relying on multiple tests. Early detection is crucial for managing the condition and preventing further decline. While language impairments are early indicators of dementia, existing speech recognition-based assessment tools primarily focus on English. This paper addresses the need for a dementia assessment system specifically for Mandarin speakers, utilizing the picture description task, a common method for evaluating cognitive and language skills.
Methods
The core of the proposed system is an attention-based, end-to-end Automatic Speech Recognition (ASR) model. This model uses a VGG network and bidirectional LSTMs for the encoder, and a double-layer LSTM with location-aware attention for the decoder. For dementia assessment, the encoder part of this ASR model is repurposed. A linear layer is added on top of the encoder, followed by a Sigmoid activation for classification (dementia vs. non-dementia) or directly for regression (predicting dementia severity scores).
The authors explore two transfer learning strategies:
- Freezing most layers: Only the last layer of the bidirectional LSTM and the final linear layer are trained.
- Unfreezing more layers: The first convolutional layer, the last bidirectional LSTM layer, and the final linear layer are trained.
They also compare their ASR-based approach with a HuBERT-based system and traditional SVMs using pre-extracted acoustic features.
Datasets
- Common Voice Corpus 11.0 (Mandarin - Taiwan): A large, publicly available dataset used for pre-training the ASR model. It provided approximately 113 hours of recordings.
- Lu Corpus: A dataset containing picture description tasks from 51 participants with a Taiwanese Mandarin accent, likely including individuals with varying degrees of dementia or aphasia. This was used for fine-tuning the ASR model.
- Local Dataset: Collected from 99 participants at Shuang-Ho Hospital. This dataset includes Mandarin speech from picture description tasks and clinical assessments (MMSE, CDR, CDR-SOB). This dataset was used for training and evaluating the dementia detection and rating prediction models.
Key Insights and Ideas
- Transfer Learning is Key: Leveraging large pre-trained ASR models (like Common Voice) significantly improves performance, especially when combined with domain-specific data (Lu Corpus and local dataset).
- Picture Description Task Effectiveness: The picture description task, when analyzed through speech, proves to be a valuable tool for identifying cognitive impairments associated with dementia.
- Temporal Information Matters: The proposed model's ability to integrate temporal information from speech patterns, text, and intonation is crucial for its superior performance compared to traditional feature-based methods.
- Unfreezing Layers Improves Adaptation: Unfreezing earlier layers of the encoder (first convolutional and last LSTM) during fine-tuning helps the model adapt better to the nuances of elderly speech and potential variations in accents or contexts, leading to improved accuracy.
Key Results
- Alzheimer's Disease Detection: The ASR-based dementia detection system achieved an accuracy of 92.04% using 5-fold cross-validation.
- Dementia Severity Prediction: The system achieved a mean absolute error (MAE) of 9% for predicting the Clinical Dementia Rating (CDR) score.
- ASR Performance: Character Error Rate (CER) for the ASR component was reduced from 253.35% to 85.21% after training with the Lu corpus.
- Comparison: The proposed ASR-based model outperformed traditional SVMs using pre-extracted acoustic features. The HuBERT-based system also showed competitive results.
Limitations and Future Work
- Dataset Size: The authors acknowledge that the limited size of the local dataset necessitated a 5-fold cross-validation approach.
- Accent Variation: While the model is tailored for Taiwanese Mandarin, further research could explore its robustness across a wider range of Mandarin accents.
- Future Directions: The authors suggest that their model can be further utilized for monitoring dementia progression and generating cognitive assessment scores. They also highlight the potential of using the same model without transfer learning, achieving a respectable 88.6% accuracy, indicating its direct applicability with patient diagnostic recordings.
Summary updated