TL;DR
This study systematically investigates the accuracy of wearable optical heart rate (HR) sensors across different skin tones and activity levels. While previous anecdotal evidence suggested skin tone might be a significant factor, this research found no statistically significant differences in HR accuracy across the Fitzpatrick skin tone scale. Instead, device type and activity level were the primary drivers of inaccuracy, with errors increasing significantly during physical activity.
What Problem Does This Paper Address?
Wearable devices are increasingly used for health monitoring, clinical research, and digital biomarker development. However, concerns about their accuracy, particularly concerning heart rate (HR) measurements derived from photoplethysmography (PPG), persist. Anecdotal evidence and some studies suggest that factors like skin tone and motion artifacts can lead to inaccurate readings. Companies are responsible for reporting accuracy, but the methods are often inconsistent and lack transparency. This lack of clear understanding of error sources hinders the equitable and reliable use of these devices in healthcare and research, especially for diverse populations. This paper aims to systematically evaluate HR accuracy across a full spectrum of skin tones and various activity conditions to identify and quantify these sources of error.
Methods
The researchers employed a mixed-effects modeling approach to analyze the data, accounting for repeated measurements from participants. They collected HR data from six different wearable devices (four consumer-grade: Apple Watch 4, Fitbit Charge 2, Garmin Vivosmart 3, Xiaomi Miband 3; and two research-grade: Empatica E4, Biovotion Everion) and compared them against a gold-standard electrocardiogram (ECG). Participants underwent a protocol involving seated rest, paced deep breathing, physical activity (walking), seated rest (washout), and a typing task.
Key analytical techniques included:
- Mean Directional Error (MDE) and Mean Absolute Error (MAE) to quantify the difference between wearable HR and ECG HR.
- Mixed-effects models to assess the impact of skin tone, activity condition, and device type on HR measurement error, including interaction effects.
- Data missingness analysis to quantify instances where wearables failed to provide HR data.
- Heart Rate Variability (HRV) analysis was performed on a subset of data (Empatica E4) to explore accuracy in this related metric.
- Signal alignment techniques (rolling window smoothing and cross-correlation) were used to mitigate potential lag issues between ECG and PPG signals.
Datasets
The study utilized data collected from 53 participants (32 females, 21 males, ages 18-54) with an equal distribution across the six Fitzpatrick (FP) skin tone categories. The participants were recruited from Duke University. The data collected included HR readings from six wearable devices and simultaneous ECG readings. No new benchmark datasets were introduced; rather, existing devices were evaluated against a gold-standard.
Key Insights and Ideas
- Skin tone is not the primary culprit: Contrary to some anecdotal claims, this study found no statistically significant difference in HR accuracy across the full spectrum of skin tones. This is a crucial finding for ensuring equitable use of wearables.
- Activity matters, a lot: The most significant source of error was activity level. HR measurements were substantially less accurate during physical activity compared to rest.
- Device differences are substantial: Not all wearables are created equal. There were significant variations in accuracy between different device models and between consumer-grade and research-grade devices.
- Over-reporting during activity: Wearables tended to over-report HR during low-intensity physical activity. This could be a built-in safety mechanism to prevent users from exceeding perceived maximum HR, but it's important for clinicians to be aware of this bias.
- Research-grade vs. Consumer-grade: Interestingly, research-grade wearables, while offering raw data for advanced analysis, were generally less accurate than consumer-grade devices in this study.
Key Results
- No significant difference in accuracy across skin tones: The study found no statistically significant correlation between skin tone and HR measurement error.
- Activity significantly increases error: Absolute error during activity was, on average, 30% higher than during rest.
- Device variability: Significant differences in accuracy were observed between devices. For example, at rest, the Apple Watch 4 had the lowest MAE among consumer devices, while the Biovotion Everion had the highest MAE among research-grade devices.
- Research-grade devices less accurate at rest: At rest, consumer-grade wearables had an average MAE of 7.2 ± 5.4 bpm, while research-grade wearables had an MAE of 13.9 ± 7.8 bpm.
- Motion artifacts are a major challenge: Rhythmic movements like walking and typing led to significantly higher errors, indicating the impact of motion artifacts on PPG signal quality.
- Data missingness: While not directly linked to skin tone, missingness varied by device and activity, with some devices showing improved data completeness during activity.
Limitations and Future Work
- Limited scope of devices: The study evaluated a specific set of devices. Future work could expand to include a wider range of wearables.
- Cost and market factors: While the paper notes that higher cost, more recent release, and larger market share might correlate with higher accuracy, the limited scope of devices prevented a definitive analysis of these factors. Further research could disentangle these influences.
- HRV analysis limited: HRV analysis was only feasible for the Empatica E4 due to data availability. Expanding HRV analysis to more devices would be valuable.
- Signal alignment complexity: While signal alignment was addressed, the dynamic nature of lags and potential processing artifacts could still influence results.
- Future work: The authors suggest that algorithms for digital biomarkers should account for these identified sources of error (device type, activity level) and that interpretation of digital biomarker data must consider this data quality. They also emphasize the need for researchers and clinicians to be aware of these biases when making clinical assessments or drawing study conclusions.
Summary updated