Subject Harmonization of Multi‐Modal Digital Markers: Improved Detection of Mild Cognitive Impairment Using Language and Facial Expression

Bao Hoang, Yijiang Pang, Hiroko H. Dodge, Jiayu ZhouAlzheimer 2024 · DOI

TL;DR

This paper explores using multi-modal digital markers (language and facial expressions) to detect Mild Cognitive Impairment (MCI). They found that harmonizing language features alone significantly improves detection performance, suggesting that careful modality-specific preprocessing is key for effective multi-modal analysis in MCI detection.

What Problem Does This Paper Address?

Early detection of Mild Cognitive Impairment (MCI) is crucial for timely intervention and effective clinical trials for Alzheimer's Disease (AD). While digital markers derived from verbal communication (language features) have shown promise, researchers are exploring how to enhance their predictive power. Specifically, this paper addresses the challenge of integrating different types of digital markers, such as language and facial expressions, which often have very distinct feature distributions. The core problem is how to effectively combine these diverse data streams to improve MCI detection, given that simply concatenating raw features might not yield optimal results.

Methods

The researchers employed a subject harmonization approach. This technique aims to remove information that is specific to individual subjects, thereby allowing the model to focus on patterns that are more generalizable and indicative of MCI. They extracted two main types of features:

  • Language Features: A 99-dimensional set derived from four categories: Linguistic Inquiry and Word Count (LIWC), Syntactic Complexity, Lexical Diversity, and Response Length.
  • Emotion Features: A 7-dimensional set representing emotions (angry, disgust, fear, happy, sad, surprise, neutral) extracted from facial frames using the DeepFace library.

These features were then used to train a neural network for MCI detection. The study systematically evaluated different combinations of harmonizing language and/or emotion features, or not harmonizing them at all, to assess their impact on classification performance.

Datasets

The study utilized data from the Internet-Based Conversational Engagement Clinical Trial (I-CONECT) (NCT02871921). This dataset comprised 3,501 conversations from 69 participants. No new benchmarks were introduced; the focus was on analyzing existing data with novel preprocessing techniques.

Key Insights and Ideas

  • Emotion adds value: Facial expression-derived emotion features can indeed augment language-based markers for MCI detection, suggesting that a multi-modal approach is beneficial.
  • Harmonization is a double-edged sword: While subject harmonization can improve performance, its effectiveness is modality-dependent. Simply harmonizing everything together isn't the optimal strategy.
  • Language harmonization shines: Harmonizing language features alone significantly boosted MCI detection performance, indicating that removing subject-specific linguistic variations is particularly impactful.
  • Modality-specific preprocessing is key: The findings strongly suggest that the harmonization strategy should be carefully selected for each individual modality rather than applied uniformly to concatenated features.

Key Results

The study evaluated performance using the Area Under the Curve (AUC) for subject classification.

  • When only language markers were used, applying subject harmonization increased detection performance.
  • When both language and emotion features were harmonized, performance decreased.
  • The best performance was achieved when language features were harmonized, but emotion features were not. This combination yielded an AUC of 0.701 ± 0.121.
  • For comparison, non-harmonized language features alone resulted in an AUC of 0.642 ± 0.134.

Limitations and Future Work

Limitations:

  • The study involved a relatively small number of participants (69), which might limit the generalizability of the findings.
  • The paper doesn't explicitly detail the neural network architecture used, which could be a factor in performance.

Future Work:

The authors suggest that future research should focus on exploring more sophisticated multi-modal fusion techniques and carefully selecting harmonization strategies for individual modalities. Further investigation into how different harmonization methods impact various types of digital biomarkers is also warranted.

Summary updated

Abstract

Abstract Background Mild Cognitive Impairment (MCI) is the prodromal stage of dementia, including Alzheimer’s Disease (AD). Early identification and accurate assessment of MCI are critical for clinical trial enrichment as well as the early intervention of AD. Digital makers offered a unique opportunity for ecologically valid and affordable early detection approaches. Language markers extracted from verbal communications have shown diagnostic efficacy in detecting early MCI. Recent studies have shown that in addition to semantic and syntactic information in dialogues, emotions in communication can also be helpful in early MCI detection. A joint analysis of language markers and emotion indicative of facial expression is thus of great interest. Features from emotion could have additional predictive benefits to language markers. One general challenge of digital biomarkers is that feature distributions are very distinct. We hereby conduct a multi‐modal analysis of language and facial expression, combined with different harmonization. Method We used 3,501 conversations from 69 participants from the Internet‐Based Conversational Engagement Clinical Trial (I‐CONECT) (NCT02871921) and extracted language and emotion variables. For language variables, we used 99‐dimensional features from four types: Linguistic Inquiry and Word Count (LIWC) (64), Syntactic Complexity (23), Lexical Diversity (10), and Response Length (2). For emotion features, we extracted a 7‐dimensional emotion feature (angry, disgust, fear, happy, sad, surprise, neutral) from a facial frame using the DeepFace library. For analysis, we used the subject harmonization approach to remove information that is predictive of subjects. The harmonized features are then used to build a neural network for MCI detection. Result We evaluated Area‐under‐Curve (AUC) on subject classification. Our results show that applying harmonization will increase detection performance when only using language markers. If we harmonize both language and emotion features, the performance decreases. However, if we do harmonization on language and without harmonization on emotion, then the performance of subject classification increases to 0.701 AUC. Conclusion Our study has shown the additional benefits of emotion variables to augment language variables in the early detection of MCI. Multimodality analyses need careful selection of harmonization: harmonization strategy should be chosen for individual modality rather than the concatenated feature variables.
BibTeX
@inproceedings{hoang2024subject,
  author = {Bao Hoang and Yijiang Pang and Hiroko H. Dodge and Jiayu Zhou},
  title = {{Subject Harmonization of Multi‐Modal Digital Markers: Improved Detection of Mild Cognitive Impairment Using Language and Facial Expression}},
  booktitle = {Alzheimer},
  year = {2024},
  doi = {10.1002/alz.094211},
}
Figure