Developing better digital health measures of Parkinson’s disease using free living data and a crowdsourced data analysis challenge

Solveig K. Sieberts, Henryk Borzymowski, Yuanfang Guan, Yidi Huang, Ayala Matzner, Alex Page, Izhar Bar‐Gad, Brett K. Beaulieu‐Jones, Yuval El-Hanani, Jann Goschenhofer, Monica Javidnia, Mark S. Keller, Yan Chak Li, Charles S. Venuto, Mohammed Saqib, Greta Smith, Ana Stanescu, Robbie Zielinski, Arun Jayaraman, Luc J. W. Evers, Luca Foschini, Alex Mariakakis, Gaurav Pandey, Nicholas Shawen, Phil Synder, Larsson OmbergmedRxiv 2021 · DOI

Abstract

One of the promising opportunities of digital health is its potential to lead to more holistic understandings of diseases by interacting with the daily life of patients and through the collection of large amounts of real-world data. Validating and benchmarking indicators of disease severity in the home setting is difficult, however, given the large number of confounders present in the real world and the challenges in collecting ground truth data in the home. Here we leverage two datasets with continuous wrist-worn accelerometer data coupled with frequent symptom reports in the home setting to develop digital biomarkers of symptom severity. Using these data, we performed a public benchmarking challenge in which participants were asked to build measures of severity across three symptoms (on/off medication, dyskinesia, and tremor). Forty-two teams participated, and performance was improved over baseline models for each subchallenge. Additional ensemble modeling across submissions further improved performance, and the top models validated in a subset of patients whose symptoms were observed and rated by trained clinicians.

Introduction: The "Snapshot" Problem in Parkinson’s Care

Parkinson’s Disease (PD) is characterized by its "flickery" nature. Symptoms such as tremors, bradykinesia, and dyskinesia fluctuate throughout the day, influenced by medication cycles, stress, and physical activity. Traditional clinical assessments often fail to capture this variability because they are mere "snapshots"—brief, infrequent visits to a clinic that cannot represent the full lived experience of a patient.

The "holy grail" of PD management is continuous, objective monitoring within the patient's home environment. However, the "real world" is computationally messy. Distinguishing a pathological tremor from routine activities like brushing teeth, or identifying involuntary dyskinesia amidst normal voluntary movement, presents a significant signal-processing challenge. Furthermore, obtaining reliable "ground truth" labels in a home setting is difficult, as it typically relies on subjective patient diaries. This research seeks to bridge the gap between noisy wearable sensor data and clinically meaningful measures of disease severity.

The BEAT-PD DREAM Challenge

To address these challenges, the authors organized the BEAT-PD DREAM Challenge, a global crowdsourced competition. Forty-two teams competed to develop the most accurate predictive models across three specific subchallenges (SC):

  1. SC1: Medication Status: Predicting whether a patient is "On" or "Off" medication.
  2. SC2: Dyskinesia Severity: Measuring the presence and intensity of involuntary movements.
  3. SC3: Tremor Severity: Quantifying the severity of Parkinsonian tremors.

Data Sources and Ground Truth

The study utilized two primary datasets consisting of continuous wrist-worn accelerometer data:

  • CIS-PD (Clinician Input Study): Data from 16 subjects using Apple Watches, providing 2,476 symptom reports.
  • REAL-PD (Parkinson@Home): Data from 12 subjects using Motorola Watches and Android phones, providing 782 symptom reports.

Ground truth labels were primarily derived from "Hauser diaries," where patients recorded their symptoms every 30 minutes. To ensure clinical relevance, a separate validation set of 1,277 segments was used, where patients performed specific tasks while being rated on a 0–4 scale by trained clinicians.

Methodological Approaches

Feature Engineering vs. Deep Learning

Participants employed a variety of technical strategies. Most top-performing teams utilized traditional signal processing to extract features—such as spectral power or signal mean—which were then fed into Random Forest classifiers. However, innovation was also seen in deep learning; one co-winning team utilized a 1D Convolutional Neural Network (CNN) to process raw sensor data directly, bypassing manual feature extraction.

Personalization and Ensembling

A key insight from the challenge was the importance of "subject-specific" models. Because PD manifests uniquely in every individual, models that learned a specific patient’s baseline often outperformed "global" models.

Following the challenge, the organizers implemented a Reinforcement Learning (RL)-based selection algorithm to create an ensemble of the top five teams. This "meta-model" learned how to weight different predictions dynamically, ultimately achieving superior accuracy compared to any single individual model.

Key Results and Insights

The models significantly outperformed the "Null" baseline (a model that simply predicts a patient's historical average) across all categories:

  • Medication Status (SC1): Team dbmi achieved a weighted Mean Square Error (MSE) of 0.878 (compared to a 0.967 null).
  • Dyskinesia (SC2): Team ROC BEAT-PD achieved an MSE of 0.405 (compared to a 0.437 null).
  • Tremor (SC3): Team yuanfang.guan achieved an MSE of 0.400 (compared to a 0.440 null).
  • Ensemble Performance: The RL-based ensemble further refined these results, reaching an MSE of 0.3937 for tremors.

The Complexity of Dyskinesia

Predicting dyskinesia proved significantly more difficult than predicting tremors. Dyskinesia often mimics normal, purposeful movement, making it harder for algorithms to isolate. The most successful models for dyskinesia relied heavily on "mean acceleration," essentially capturing the overall "busyness" or excess of movement.

Clinical Validation

Crucially, models trained on subjective patient-reported diaries successfully correlated with objective clinician-rated assessments. This finding validates that digital biomarkers are not merely capturing a patient's subjective feeling but are tracking the underlying objective pathology of the disease.

Limitations and Future Directions

While the study represents a major proof-of-concept, several hurdles remain before these measures are clinically actionable:

  • Sample Size: Although the data was longitudinal, the total number of subjects (16–22) was small. Thousands of subjects may be required to build a truly "global" model that does not require individual tuning.
  • Activity Context: Current models struggle to distinguish symptoms from specific activities (e.g., walking vs. tremor). Future iterations should integrate Human Activity Recognition (HAR) to provide necessary context to the sensor data.
  • Label Bias: Patient diaries are prone to recall bias. Moving toward "ecological momentary assessments" (EMAs) or clinician-labeled home video could provide higher-quality ground truth.

Conclusion

By leveraging crowdsourced expertise and ensemble modeling, this research demonstrates that passive data from consumer wearables can effectively monitor Parkinson’s symptoms in the messy environment of daily life. This moves the field closer to a future where PD treatment can be personalized and adjusted based on continuous, objective data rather than infrequent clinical snapshots.

Summary updated

Abstract

Abstract One of the promising opportunities of digital health is its potential to lead to more holistic understandings of diseases by interacting with the daily life of patients and through the collection of large amounts of real world data. Validating and benchmarking indicators of disease severity in the home setting is difficult, however, given the large number of confounders present in the real world and the challenges in collecting ground truth data in the home. Here we leverage two datasets with continuous wrist-worn accelerometer data coupled with frequent symptom reports in the home setting, to develop digital biomarkers of symptom severity. Using these data, we performed a public benchmarking challenge in which participants were asked to build measures of severity across 3 symptoms (on/off medication, dyskinesia, and tremor). 42 teams participated and performance was improved over baseline models for each subchallenge. Additional ensemble modeling across submissions further improved performance, and the top models validated in a subset of patients whose symptoms were observed and rated by trained clinicians.
BibTeX
@misc{sieberts2021developing,
  author = {Solveig K. Sieberts and Henryk Borzymowski and Yuanfang Guan and Yidi Huang and Ayala Matzner and Alex Page and Izhar Bar‐Gad and Brett K. Beaulieu‐Jones and Yuval El-Hanani and Jann Goschenhofer and Monica Javidnia and Mark S. Keller and Yan Chak Li and Charles S. Venuto and Mohammed Saqib and Greta Smith and Ana Stanescu and Robbie Zielinski and Arun Jayaraman and Luc J. W. Evers and Luca Foschini and Alex Mariakakis and Gaurav Pandey and Nicholas Shawen and Phil Synder and Larsson Omberg},
  title = {{Developing better digital health measures of Parkinson’s disease using free living data and a crowdsourced data analysis challenge}},
  year = {2021},
  doi = {10.1101/2021.10.20.21265298},
  howpublished = {medRxiv},
}
Figure