Multimodal characterization of Alzheimer’s Disease using speech, eye movement, and handwriting

Laureano Moro-Velázquez, Anna Favaro, Thomas Thebaud, Yuzhe Wang, Ankur Butala, Esther S. Oh, Najim DehakAlzheimer 2024 · DOI

TL;DR

This paper explores using a combination of speech, handwriting, and eye movement data to better characterize Alzheimer's Disease (AD). By fusing these three digital biomarkers, they achieved a slightly higher accuracy in distinguishing AD patients from healthy controls compared to using any single modality alone, suggesting a more comprehensive approach to AD phenotyping.

What Problem Does This Paper Address?

Accurately identifying and characterizing Alzheimer's Disease (AD) is crucial for personalized treatment. While individual digital biomarkers, like speech patterns, have shown promise in detecting AD, relying on a single source of information might miss subtle cues. This study addresses the gap in understanding how combining multiple, diverse digital biomarkers—specifically speech, handwriting, and eye movements—can lead to a more robust and insightful characterization of individuals in the early stages of AD compared to cognitively normal (CN) individuals.

Methods

The researchers employed a multimodal approach, collecting data from participants performing specific tasks. Speech was captured while participants read a passage aloud, and eye movements were simultaneously recorded using a high-precision eye-tracker. Handwriting data was gathered as participants drew a spiral on a digital pad. They then extracted a comprehensive set of metrics: 57 for speech (articulation, prosody, hesitation), 90 for handwriting (drawing entropy, speed, acceleration), and 32 for eye movement (saccades, fixations, blinks). Multilayer Perceptron (MLP) classifiers were trained for each modality individually and for a combined, early-fusion multimodal model. To understand the contribution of each metric, SHAP (SHapley Additive exPlanations) values were calculated to rank their impact on the classification.

Datasets

The study utilized data collected from two groups: 11 participants diagnosed with early-stage AD or mild cognitive impairment due to AD, and 13 age-matched cognitively normal (CN) participants. This forms a relatively small, specialized dataset for evaluating the multimodal approach.

Key Insights and Ideas

  • Synergy of Modalities: The core idea is that combining different behavioral signals (what you say, how you write, and where you look) can provide a more complete picture of cognitive function than any single signal alone.
  • Complementary Information: Each modality captures different aspects of cognitive processing. Speech reflects language production and fluency, handwriting can reveal fine motor control and cognitive planning, and eye movements offer insights into attention and visual processing.
  • Handwriting's Prominence: Interestingly, handwriting metrics emerged as the most influential in the multimodal model, suggesting that subtle changes in drawing or writing patterns might be particularly sensitive indicators of early AD.

Key Results

  • Multimodal Advantage: The multimodal classifier achieved a cross-validation accuracy of 77.9%, a slight but notable improvement over the best single modalities, which ranged from 75.3% (speech) to 77.4% (handwriting).
  • Top Performing Metrics: Within individual modalities, "drawing velocity skewness" showed the highest discriminative power for handwriting (AUC=0.83), "pause variability" for speech (AUC=0.79), and "number of fixations" for eye movement (AUC=0.69).
  • Handwriting Dominance in Multimodal Model: SHAP analysis indicated that handwriting metrics had the most significant impact on the overall performance of the combined multimodal classifier.

Limitations and Future Work

The study acknowledges its relatively small sample size (11 AD/MCI, 13 CN participants), which limits the generalizability of the findings. Future work could involve validating these findings on larger and more diverse cohorts. Additionally, exploring different fusion strategies beyond early fusion and investigating the longitudinal changes in these multimodal biomarkers over time would be valuable for tracking disease progression and treatment response.

Summary updated

Abstract

Abstract Background Phenotyping Alzheimer’s Disease (AD) can be crucial to providing personalized treatment. Several studies have analyzed the use of digital biomarkers to characterize a subject’s behavior, usually obtained from a single modality, such as speech. However, combining several modalities in a single study has not been deeply studied. We aim to characterize people with AD and differentiate them from a cognitively normal control group (CN), employing metrics obtained through from three different modalities: speech, handwriting, and eye movement. Method We collected data from a group of 11 participants in the early stages of AD or with, mild cognitive impairment due to AD, and an age‐matched CN group of 13 participants. Each participant read out loud the Rainbow Passage, while their speech was recorded with a headset microphone and their eye movement with a high precision eye‐tracker, simultaneously. Then, the participants were asked to draw a spiral on a digital pad. We obtained 57 speech metrics, related to articulation, prosody, and hesitation; 90 handwriting metrics, characterizing drawing entropy, writing speed, and acceleration; 32 eye movement metrics related to saccade, fixation, and blink statistics. Several multilayer perceptron (MLP) classifiers were trained per modality and using all modalities together, following a cross‐validation strategy. SHAP values were calculated to rank metric impact in the differentiation between AD and CN. Result Cross‐validation accuracy in the binary classification of AD/CN ranged between 75.3% and 77.4% for single modalities, whereas the combination of the three modalities provided a 77.9% accuracy. The drawing velocity skewness metric provided the highest area under the receiver operating characteristic curve (AUC) from all the handwriting features (0.83), whereas the pause variability was the speech metric with the highest AUC (0.79). The number of fixations provided the best differentiation between groups in the eye movement modality, with AUC = 0.69. Conclusion Combining the three proposed modalities provides a higher differentiation power than using each modality separately. The combination can provide a more insightful and complementary characterization of patients. Handwriting metrics have the most impact on the prediction of the multimodal MLP model, according to the SHAP values on the multimodal classifier.
BibTeX
@inproceedings{morovelazquez2024multimodal,
  author = {Laureano Moro-Velázquez and Anna Favaro and Thomas Thebaud and Yuzhe Wang and Ankur Butala and Esther S. Oh and Najim Dehak},
  title = {{Multimodal characterization of Alzheimer’s Disease using speech, eye movement, and handwriting}},
  booktitle = {Alzheimer},
  year = {2024},
  doi = {10.1002/alz.094195},
}
Figure