Abstract
One of the promising opportunities of digital health is its potential to lead to more holistic understandings of diseases by interacting with the daily life of patients and through the collection of large amounts of real-world data. Validating and benchmarking indicators of disease severity in the home setting is difficult, however, given the large number of confounders present in the real world and the challenges in collecting ground truth data in the home. Here we leverage two datasets with continuous wrist-worn accelerometer data coupled with frequent symptom reports in the home setting to develop digital biomarkers of symptom severity. Using these data, we performed a public benchmarking challenge in which participants were asked to build measures of severity across three symptoms (on/off medication, dyskinesia, and tremor). Forty-two teams participated, and performance was improved over baseline models for each subchallenge. Additional ensemble modeling across submissions further improved performance, and the top models validated in a subset of patients whose symptoms were observed and rated by trained clinicians.
Introduction: The "Snapshot" Problem in Parkinson’s Care
Parkinson’s Disease (PD) is characterized by its "flickery" nature. Symptoms such as tremors, bradykinesia, and dyskinesia fluctuate throughout the day, influenced by medication cycles, stress, and physical activity. Traditional clinical assessments often fail to capture this variability because they are mere "snapshots"—brief, infrequent visits to a clinic that cannot represent the full lived experience of a patient.
The "holy grail" of PD management is continuous, objective monitoring within the patient's home environment. However, the "real world" is computationally messy. Distinguishing a pathological tremor from routine activities like brushing teeth, or identifying involuntary dyskinesia amidst normal voluntary movement, presents a significant signal-processing challenge. Furthermore, obtaining reliable "ground truth" labels in a home setting is difficult, as it typically relies on subjective patient diaries. This research seeks to bridge the gap between noisy wearable sensor data and clinically meaningful measures of disease severity.
The BEAT-PD DREAM Challenge
To address these challenges, the authors organized the BEAT-PD DREAM Challenge, a global crowdsourced competition. Forty-two teams competed to develop the most accurate predictive models across three specific subchallenges (SC):
- SC1: Medication Status: Predicting whether a patient is "On" or "Off" medication.
- SC2: Dyskinesia Severity: Measuring the presence and intensity of involuntary movements.
- SC3: Tremor Severity: Quantifying the severity of Parkinsonian tremors.
Data Sources and Ground Truth
The study utilized two primary datasets consisting of continuous wrist-worn accelerometer data:
- CIS-PD (Clinician Input Study): Data from 16 subjects using Apple Watches, providing 2,476 symptom reports.
- REAL-PD (Parkinson@Home): Data from 12 subjects using Motorola Watches and Android phones, providing 782 symptom reports.
Ground truth labels were primarily derived from "Hauser diaries," where patients recorded their symptoms every 30 minutes. To ensure clinical relevance, a separate validation set of 1,277 segments was used, where patients performed specific tasks while being rated on a 0–4 scale by trained clinicians.
Methodological Approaches
Feature Engineering vs. Deep Learning
Participants employed a variety of technical strategies. Most top-performing teams utilized traditional signal processing to extract features—such as spectral power or signal mean—which were then fed into Random Forest classifiers. However, innovation was also seen in deep learning; one co-winning team utilized a 1D Convolutional Neural Network (CNN) to process raw sensor data directly, bypassing manual feature extraction.
Personalization and Ensembling
A key insight from the challenge was the importance of "subject-specific" models. Because PD manifests uniquely in every individual, models that learned a specific patient’s baseline often outperformed "global" models.
Following the challenge, the organizers implemented a Reinforcement Learning (RL)-based selection algorithm to create an ensemble of the top five teams. This "meta-model" learned how to weight different predictions dynamically, ultimately achieving superior accuracy compared to any single individual model.
Key Results and Insights
The models significantly outperformed the "Null" baseline (a model that simply predicts a patient's historical average) across all categories:
- Medication Status (SC1): Team
dbmiachieved a weighted Mean Square Error (MSE) of 0.878 (compared to a 0.967 null). - Dyskinesia (SC2): Team
ROC BEAT-PDachieved an MSE of 0.405 (compared to a 0.437 null). - Tremor (SC3): Team
yuanfang.guanachieved an MSE of 0.400 (compared to a 0.440 null). - Ensemble Performance: The RL-based ensemble further refined these results, reaching an MSE of 0.3937 for tremors.
The Complexity of Dyskinesia
Predicting dyskinesia proved significantly more difficult than predicting tremors. Dyskinesia often mimics normal, purposeful movement, making it harder for algorithms to isolate. The most successful models for dyskinesia relied heavily on "mean acceleration," essentially capturing the overall "busyness" or excess of movement.
Clinical Validation
Crucially, models trained on subjective patient-reported diaries successfully correlated with objective clinician-rated assessments. This finding validates that digital biomarkers are not merely capturing a patient's subjective feeling but are tracking the underlying objective pathology of the disease.
Limitations and Future Directions
While the study represents a major proof-of-concept, several hurdles remain before these measures are clinically actionable:
- Sample Size: Although the data was longitudinal, the total number of subjects (16–22) was small. Thousands of subjects may be required to build a truly "global" model that does not require individual tuning.
- Activity Context: Current models struggle to distinguish symptoms from specific activities (e.g., walking vs. tremor). Future iterations should integrate Human Activity Recognition (HAR) to provide necessary context to the sensor data.
- Label Bias: Patient diaries are prone to recall bias. Moving toward "ecological momentary assessments" (EMAs) or clinician-labeled home video could provide higher-quality ground truth.
Conclusion
By leveraging crowdsourced expertise and ensemble modeling, this research demonstrates that passive data from consumer wearables can effectively monitor Parkinson’s symptoms in the messy environment of daily life. This moves the field closer to a future where PD treatment can be personalized and adjusted based on continuous, objective data rather than infrequent clinical snapshots.
Summary updated