On the analysis of personalized medication response and classification of case vs control patients in mobile health studies: the mPower case study

Elias Chaibub Neto, Thanneer M. Perumal, Abhishek Pratap, Brian M. Bot, Lara M. Mangravite, Larsson OmbergarXiv 2017 · DOI · arXiv

Abstract: In this work we provide a couple of contributions to the analysis of longitudinal data collected by smartphones in mobile health applications. First, we propose a novel statistical approach to disentangle personalized treatment and "time-of-the-day" effects in observational studies. Under the assumption of no unmeasured confounders, we show how to use conditional independence relations in the data in order to determine if a difference in performance between activity tasks performed before and after the participant has taken medication, are potentially due to an effect of the medication or to a "time-of-the-day" effect (or still to both). Second, we show that smartphone data collected from a given study participant can represent a "digital fingerprint" of the participant, and that classifiers of case/control labels, constructed using longitudinal data, can show artificially improved performance when data from each participant is included in both training and test sets. We illustrate our contributions using data collected during the first 6 months of the mPower study.

Introduction

Mobile health (mHealth) applications offer a "lab in your pocket," collecting massive amounts of longitudinal sensor data (e.g., accelerometer, voice) to monitor chronic conditions like Parkinson’s disease (PD). However, this observational data presents two major hurdles: the confounding influence of daily rhythms on medication response and the "digital fingerprint" effect that can lead to deceptive machine learning performance.

The Confounding Clock and Digital Fingerprints

In observational studies, patients decide when to take medication and when to perform tasks. If a patient consistently performs "before medication" tasks in the morning and "after medication" tasks in the afternoon, improvements might be due to the drug or simply natural circadian fluctuations in alertness and motor function.

Furthermore, mHealth data is highly unique to individuals. If a single participant contributes multiple samples that are randomly split between training and testing sets, a machine learning model may simply learn to recognize that specific person’s movement patterns (their "digital fingerprint") rather than the general features of the disease.

Methods

The authors introduce a two-pronged approach to address these issues:

  1. Causal Disentangling: Using Directed Acyclic Graphs (DAGs) to model relationships between Treatment ($X$), Time-of-day ($T$), and Outcome ($Y$). Conditional independence tests determine if medication has a direct effect on performance independent of timing.
  2. Robust Time-Series Analysis: Moving beyond naive linear regression, the study employs ARIMA (Autoregressive Integrated Moving Average) errors and Newey-West HAC covariance estimators to account for serial correlation in longitudinal data.
  3. Validation via Label Shuffling: To prove the "digital fingerprint" effect, the researchers compared Random Forest classifier performance on original labels versus "block-shuffled" labels, where participant identities are preserved but case/control status is randomized.

Datasets: The mPower Study

The study utilizes the first six months of the mPower study, a large-scale smartphone-based PD observatory. The analysis focused on subsets of participants who provided sufficient longitudinal data across three tasks: speeded tapping, voice (sustained phonation), and walk/rest (gait and balance).

Age distribution of PD cases and healthy controls across the mPower study and specific tasks (tapping, voice, walk/rest).

Key Insights and Results

The Fingerprint Artifact

The most striking finding is that classifiers can distinguish between random groups of people as easily as they distinguish PD patients from controls. When data from the same participant is allowed in both training and test sets, Area Under the Curve (AUC) values were often near-perfect (>0.9). However, when splitting the data by participant, performance dropped significantly, revealing the true, more modest signal.

Illustration of data splitting strategies: sample-level splitting (top) can lead to data leakage via digital fingerprints, while participant-level splitting (bottom) provides a more robust evaluation of disease classification.

Medication Response and Heterogeneity

Using the disentangling method, the researchers found that roughly 37% to 45% of participants showed a statistically significant response to medication in the tapping task. Notably, medication response was highly heterogeneous; some patients responded in tapping but not voice, or vice versa. Additionally, the strength of the medication response correlated with the number of years since diagnosis, aligning with clinical knowledge of "on/off" fluctuations in progressing Parkinson's.

Conclusion

While mHealth data holds great promise for individualized medicine, "in the wild" data is noisy and prone to significant confounding. Achieving clinical-grade diagnostic performance requires larger cohorts and algorithms designed to ignore individual "fingerprints" in favor of disease-specific features. Accounting for time-of-day effects is essential to avoid misattributing natural rhythms to pharmacological efficacy.

Summary updated

Abstract

In this work we provide a couple of contributions to the analysis of longitudinal data collected by smartphones in mobile health applications. First, we propose a novel statistical approach to disentangle personalized treatment and "time-of-the-day" effects in observational studies. Under the assumption of no unmeasured confounders, we show how to use conditional independence relations in the data in order to determine if a difference in performance between activity tasks performed before and after the participant has taken medication, are potentially due to an effect of the medication or to a "time-of-the-day" effect (or still to both). Second, we show that smartphone data collected from a given study participant can represent a "digital fingerprint" of the participant, and that classifiers of case/control labels, constructed using longitudinal data, can show artificially improved performance when data from each participant is included in both training and test sets. We illustrate our contributions using data collected during the first 6 months of the mPower study.
BibTeX
@misc{neto2017analysis,
  author = {Elias Chaibub Neto and Thanneer M. Perumal and Abhishek Pratap and Brian M. Bot and Lara M. Mangravite and Larsson Omberg},
  title = {{On the analysis of personalized medication response and classification of case vs control patients in mobile health studies: the mPower case study}},
  year = {2017},
  eprint = {1706.09574},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.09574},
}
Figure