Developing Measures of Cognitive Impairment in the Real World from Consumer-Grade Multimodal Sensor Streams

Richard Chen, Filip Jankovic, Nikki Marinsek, Luca Foschini, Lampros Kourtis, Alessio Signorini, Melissa Pugh, Jie Shen, Roy Yaari, Vera Maljkovic, Marc Sunga, Han Hee Song, Hyun Joon Jung, Belle L. Tseng, Andrew TristerKDD 2019 · DOI

TL;DR

This paper introduces a novel platform for passively collecting multimodal sensor data from consumer devices to detect cognitive impairment. By analyzing 16TB of data from a 12-week study, they demonstrate that these real-world behavioral signatures can effectively differentiate individuals with mild cognitive impairment (MCI) and early Alzheimer's disease (AD) from healthy controls.

What Problem Does This Paper Address?

Diagnosing cognitive impairment, especially in its early stages (like MCI), is a significant challenge. Current clinical screening tools often miss subtle early symptoms, are time-consuming, and can be influenced by rater bias or cultural factors. While computerized tests and gamified apps offer some improvements, they can be susceptible to practice effects. Purely passive measurements are ideal but often require complex, home-installed sensors, limiting scalability. This paper tackles the need for unobtrusive, scalable, and cost-effective methods to detect and monitor cognitive decline in real-world settings, bridging the gap between clinical assessments and everyday life.

Methods

The core of this work is the Evidation Study Platform, a secure, HIPAA-compliant system designed to ingest, process, and analyze high-volume, multimodal sensor data from consumer devices. Key methodological contributions include:

  • Multimodal Data Integration: The platform harmonizes data from various sources, including iPhones, Apple Watches, and Beddit sleep monitors, capturing a rich tapestry of daily activities.
  • Behaviorgrams: A novel representation that aligns and aggregates diverse data streams (event-based, time-series) into a common, minute-level resolution format. This allows for the analysis of complex behavioral patterns.
  • Intelligent Data Handling: Sophisticated techniques for time-alignment, imputation of missing data (treating missingness as a potential signal), and feature engineering tailored to different data types.
  • Machine Learning for Classification: An Extreme Gradient Boosting (XGBoost) regression model was employed, leveraging a pairwise ranking objective function to differentiate between healthy controls, MCI, and mild AD.
  • Data Augmentation: A window-slicing technique was used to create multiple non-overlapping 2-week subsets from participants' data, effectively augmenting the dataset for training.
  • Interpretable AI: SHAP (SHapley Additive explanations) values were used to understand which features were most influential in the model's predictions.

Datasets

The primary dataset used in this study was collected during the Lilly Exploratory Digital Assessment Study. This was a 12-week, IRB-approved, multi-site feasibility study involving:

  • Participants: 82 healthy controls, 24 individuals with Mild Cognitive Impairment (MCI), and 7 individuals with early Alzheimer's Disease (AD).
  • Data Collection: Over 16TB of data was collected passively from participants' personal consumer devices (iPhone, Apple Watch, Beddit) and through active tasks performed on a Digital Assessment App on an iPad.

This dataset is proprietary and not publicly available.

Key Insights and Ideas

  • Everyday Behavior as a Biomarker: The study powerfully demonstrates that subtle changes in everyday behaviors, captured by consumer-grade sensors, can serve as robust indicators of cognitive impairment. This moves beyond lab-based tests to real-world functional assessment.
  • Missing Data as a Signal: The hypothesis that gaps in data might signal cognitive impairment is a crucial insight. Instead of aggressively imputing all missing values, the researchers strategically handled them, recognizing their potential diagnostic value.
  • Multimodality is Key: Combining data from various sensors (activity, sleep, app usage, communication patterns) provides a more comprehensive picture of an individual's cognitive state than any single source alone.
  • Consumer Tech's Potential in Healthcare: This work highlights the transformative potential of widely available consumer technology for large-scale, continuous health monitoring, particularly for chronic conditions like cognitive decline.

Key Results

  • Effective Differentiation: The developed model, using device-derived features, achieved an Area Under the ROC Curve (AUROC) of 0.771 for differentiating healthy controls from symptomatic individuals (MCI/AD) at the participant level.
  • Demographics vs. Device Data: While demographics provided some predictive power (AUROC 0.757), device-derived features alone performed comparably and, when combined with demographics, improved the AUROC to 0.804. Crucially, device features alone showed better precision (AUPRC 0.628 vs 0.546 for demographics).
  • Age-Matched Performance: Even when controlling for age differences, device-derived features maintained moderate performance (AUROC 0.726), indicating they capture differences beyond normal aging.
  • Mild AD Detection: For the small subset of mild AD dementia cases, the model achieved high AUROC scores (up to 0.933 when combined with demographics), suggesting strong potential for detecting more advanced stages of the disease.
  • Feature Importance: Key features identified by SHAP analysis included typing speed, time of day of first active pace, energy survey compliance, and message frequency, all of which showed significant differences between symptomatic and healthy groups.

Limitations and Future Work

  • Feasibility Study: The study acknowledges its nature as a feasibility assessment, with a relatively small sample size for the symptomatic groups, especially for mild AD.
  • Behavioral Shifts: The authors note that some identified behavioral patterns might be modifiable and not solely indicative of disease progression. Future work needs to account for such shifts.
  • Self-Reinforcing Decline: The potential for knowledge of cognitive decline to cause further decline is a noted concern.
  • Generalizability and Bias: The authors recognize the risk of automated decision-making tools trained on data distributions that may not be representative or may shift over time.
  • Future Directions:
    • More in-depth exploration of accelerometer, audio, and video data.
  • Longer longitudinal studies with larger cohorts.
  • Developing more accurate classification models.
  • Investigating N-of-1 longitudinal settings to detect individual changes over time.
  • Addressing privacy concerns and implementing on-device control for users.
  • Exploring the use of these methods for monitoring treatment effectiveness and accelerating drug development.

Summary updated

BibTeX
@inproceedings{chen2019developing,
  author = {Richard Chen and Filip Jankovic and Nikki Marinsek and Luca Foschini and Lampros Kourtis and Alessio Signorini and Melissa Pugh and Jie Shen and Roy Yaari and Vera Maljkovic and Marc Sunga and Han Hee Song and Hyun Joon Jung and Belle L. Tseng and Andrew Trister},
  title = {{Developing Measures of Cognitive Impairment in the Real World from Consumer-Grade Multimodal Sensor Streams}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2019},
  doi = {10.1145/3292500.3330690},
}
Figure