A multimodal screening system for elderly neurological diseases based on deep learning

Sang‐Young Park, Changho No, Sora Kim, Kyoungmin Han, Jin‐Man Jung, Kyum-Yil Kwon, Minsik LeeScientific 2023 · DOI

TL;DR

This paper introduces a novel deep learning system that uses video and voice data from specific physical and verbal tasks to screen for neurological diseases like stroke and Parkinson's. By combining features from body movements, facial expressions, and speech, the system achieves promising accuracy, offering a more accessible and potentially earlier detection method than traditional medical imaging.

What Problem Does This Paper Address?

Neurological diseases, such as stroke and Parkinson's disease, are a significant global health burden, with their incidence expected to rise due to aging populations. Early diagnosis is crucial for effective treatment and reducing long-term impacts. However, current diagnostic methods like MRI and PET scans are costly and not always practical for widespread screening. The paper addresses this by proposing a non-invasive, video- and voice-based screening system that leverages deep learning to identify subtle signs of these diseases. The challenge lies in capturing and integrating diverse, often subtle, human movement and speech patterns that indicate neurological dysfunction, especially in a way that is efficient and generalizable.

Methods

The core of the proposed system is a multimodal deep neural network that integrates information from human body landmarks (extracted from video) and voice features.

  • Data Acquisition: A set of 15 specific examination protocols were designed, involving simple physical movements and speech tasks. Participants performed these protocols while being recorded by multiple 4K cameras.
  • Feature Extraction:
    • Landmarks: Human body landmarks (for body, face, and hands) were extracted from the video data. Spatiotemporal features were then captured using pre-trained Semantics-Guided Neural Networks (SGNs).
  • Voice: Mel-frequency cepstral coefficients (MFCCs) were extracted from the audio component of the recordings. These features were processed by a pre-trained Crossed-Time Delay Neural Network (CTDNN).
  • Frame-Length Staticizer: A modified frame-length staticizer was employed to handle variable video lengths without losing critical information, particularly momentary tremors. This method segments and concatenates frames to create fixed-length sequences.
  • Network Architecture:
    • Subnetworks: Each protocol's extracted features (landmarks or voice) are fed into dedicated subnetworks (SGNs or CTDNN) to extract protocol-specific high-level features.
  • Feature Aggregator: A feature aggregator then combines the outputs from all subnetworks.
  • Multitask Learning: The system is trained using multitask learning, considering both disease labels (stroke, Parkinson's, or healthy control) and protocol labels (indicating if a specific task was performed normally). This allows the model to learn from both disease-specific and task-specific performance.
  • Pre-training: The SGNs and CTDNN were pre-trained on large, publicly available datasets (NTU RGB+D60 for body, DAISEE for face, NVGesture for hands, and VoxCeleb for voice) to leverage existing knowledge and improve feature extraction.

Datasets

  • KSSP Dataset: A newly collected dataset specifically for this study, comprising video and voice recordings of 574 subjects (125 patients with stroke or Parkinson's disease, and 449 healthy controls) performing 15 distinct examination protocols. Data was collected from two hospitals (Korea University Ansan Hospital and Soonchunhyang University Seoul Hospital).
  • Pre-training Datasets:
    • NTU RGB+D60: Used for pre-training SGNs for body landmarks.
  • DAISEE: Used for pre-training SGNs for face landmarks.
  • NVGesture: Used for pre-training SGNs for hand landmarks.
  • VoxCeleb: Used for pre-training CTDNN for voice features.

Key Insights and Ideas

  • Multimodality is Key: Combining voice and body landmark data significantly improves screening performance compared to using either modality alone, highlighting the complementary nature of these signals for detecting neurological deficits.
  • Protocol Design Matters: The choice of examination protocols is critical. Protocols involving more dynamic body movements (for stroke) or specific speech tasks (for both stroke and Parkinson's) showed higher diagnostic power.
  • Face Landmarks for Parkinson's, Body for Stroke: The study suggests that facial landmarks are particularly effective for screening Parkinson's disease, while body movements are more indicative of stroke.
  • Generalization through Multimodality: While single modalities might show high validation AUC, the multimodal approach demonstrates better generalization to unseen test data, suggesting it's more robust.
  • Video Cameras are Sufficient: The system's reliance on readily available video cameras and microphones makes it a potentially accessible and cost-effective screening tool.

Key Results

The proposed multimodal system achieved the following test AUC scores:

  • Stroke: 0.802
  • Parkinson's Disease: 0.780

These results are considered effective for a screening system. The study also found that using a combination of center and left cameras yielded the best performance for landmark data, suggesting that multiple views can capture crucial information while avoiding overfitting issues associated with using too many cameras.

Limitations and Future Work

  • Dataset Size and Diversity: While the KSSP dataset is substantial, the authors acknowledge that larger datasets and more diverse patient populations would be beneficial for further improving the model's robustness and generalizability.
  • Privacy Constraints: The KSSP dataset is not publicly available due to privacy concerns, which limits broader research collaboration.
  • Attention Mechanisms: The authors suggest exploring more advanced network architectures, such as those incorporating attention mechanisms, to automatically learn the importance of different protocols and features.
  • Broader Disease Scope: Future work could extend the system to screen for a wider range of neurological diseases.
  • Severity Prediction: Predicting the severity of diagnosed diseases is identified as an area for future research.

Summary updated

Abstract

In this paper, we propose a deep-learning-based algorithm for screening neurological diseases. We proposed various examination protocols for screening neurological diseases and collected data by video-recording persons performing these protocols. We converted video data into human landmarks that capture action information with a much smaller data dimension. We also used voice data which are also effective indicators of neurological disorders. We designed a subnetwork for each protocol to extract features from landmarks or voice and a feature aggregator that combines all the information extracted from the protocols to make a final decision. Multitask learning was applied to screen two neurological diseases. To capture meaningful information about these human landmarks and voices, we applied various pre-trained models to extract preliminary features. The spatiotemporal characteristics of landmarks are extracted using a pre-trained graph neural network, and voice features are extracted using a pre-trained time-delay neural network. These extracted high-level features are then passed onto the subnetworks and an additional feature aggregator that are simultaneously trained. We also used various data augmentation techniques to overcome the shortage of data. Using a frame-length staticizer that considers the characteristics of the data, we can capture momentary tremors without wasting information. Finally, we examine the effectiveness of different protocols and different modalities (different body parts and voice) through extensive experiments. The proposed method achieves AUC scores of 0.802 for stroke and 0.780 for Parkinson's disease, which is effective for a screening system.
BibTeX
@inproceedings{park2023multimodal,
  author = {Sang‐Young Park and Changho No and Sora Kim and Kyoungmin Han and Jin‐Man Jung and Kyum-Yil Kwon and Minsik Lee},
  title = {{A multimodal screening system for elderly neurological diseases based on deep learning}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-48071-y},
}
Figure