Heart Rate Measurement Accuracy of Fitbit Charge 4 and Samsung Galaxy Watch Active2: Device Evaluation Study (Preprint)

Michael Nissen, Syrine Slim, Katharina Jäger, Madeleine Flaucher, Hanna Huebner, Nina Danzberger, Peter A. Fasching, Matthias W. Beckmann, Stefan Gradl, Bjoern M. Eskofier2021 · DOI

TL;DR

This study evaluates the heart rate accuracy of the Fitbit Charge 4 and Samsung Galaxy Watch Active2 against a medical-grade ECG. Both devices generally achieved acceptable accuracy (mean absolute percentage error <10%) across various activities, making them potentially suitable for research, though neither was accurate enough for seated rest or typing.

What Problem Does This Paper Address?

Wearable devices like smartwatches and fitness trackers are increasingly used in medical research for their ability to collect continuous, real-world data. However, their utility hinges on their accuracy. Before these devices can be reliably integrated into studies, their performance needs rigorous validation against established medical standards. This paper addresses the need to validate two popular, yet previously unvalidated, consumer wearables – the Fitbit Charge 4 and Samsung Galaxy Watch Active2 – for their heart rate measurement accuracy in a controlled setting with diverse activities.

Methods

The researchers recruited 23 healthy adults to participate in a study protocol involving 10 distinct activities, ranging from resting and sedentary tasks (e.g., seated rest, keyboard typing) to low-intensity (e.g., standing up, slow walking) and high-intensity exercises (e.g., brisk walking, climbing stairs, squats). Participants wore both the Fitbit Charge 4 and the Samsung Galaxy Watch Active2 simultaneously, with a medically certified Holter ECG serving as the gold standard for reference heart rate data.

Data from all three devices were synchronized and compared at 1-second intervals. The study analyzed several metrics to assess accuracy, including:

  • Mean Absolute Error (MAE): The average absolute difference between the device and reference heart rate.
  • Mean Absolute Percentage Error (MAPE): The average percentage difference.
  • Lin Concordance Correlation Coefficient (CCC): A measure of agreement between the two measurements.
  • Pearson Correlation Coefficient (PCC): A measure of linear correlation.
  • Bland-Altman plots: Visualizations to assess the agreement and identify bias.

Datasets

The study did not use pre-existing datasets for training. Instead, it generated a new dataset by collecting simultaneous heart rate measurements from 23 participants using the Fitbit Charge 4, Samsung Galaxy Watch Active2, and a Mind Media NeXus-10 MKI Holter ECG. This dataset was used for evaluation.

Key Insights and Ideas

  • Activity-Dependent Accuracy: The accuracy of both wearables varied significantly depending on the type of activity. This highlights the importance of considering the specific use case when selecting a wearable for research.
  • No One-Size-Fits-All: Neither device was consistently superior across all activities. The Fitbit Charge 4 performed better in resting and sedentary conditions, while the Samsung Galaxy Watch Active2 was more accurate during higher-intensity exercises.
  • Limitations of Consumer Devices: Even with a MAPE below 10% (often considered acceptable for research), neither device achieved sufficient accuracy for critical resting or typing activities, suggesting that researchers must be cautious about using them for these specific scenarios.
  • Data Quality Challenges: The study encountered issues with noisy ECG data during high-intensity activities (squats, stair climbing) and also noted peculiar HR readings from the Samsung device in one participant, underscoring the challenges of data collection and the need for robust data cleaning and annotation.

Key Results

  • Overall Accuracy: Both devices achieved a Mean Absolute Percentage Error (MAPE) of less than 10% across all activities, generally meeting the threshold for acceptable accuracy in research.
  • Fitbit Charge 4: Slightly underestimated heart rate overall (-1.66 bpm bias). It showed lower error rates during resting and sedentary activities. However, it exhibited a particularly high bias during standing up (-7.95 bpm) and squats (-12.52 bpm).
  • Samsung Galaxy Watch Active2: Slightly overestimated heart rate overall (3.84 bpm bias). It performed better during and after low- and high-intensity activities. Its highest biases were observed during typing (8.63 bpm) and laying down on the left side (6.01 bpm).
  • Specific Activity Performance:
    • Resting/Sedentary: Fitbit Charge 4 had lower MAE.
  • Low/High Intensity: Samsung Galaxy Watch Active2 was more accurate.
  • Seated Rest & Typing: Neither device achieved sufficient accuracy for these activities.
  • Data Exclusion: Data from 3 participants were completely excluded, and data from 2 participants were excluded for specific activities due to issues like electrode detachment or excessive noise in the criterion device.

Limitations and Future Work

  • Participant Demographics: The study primarily included young, healthy participants with a specific range of Fitzpatrick skin types. This may limit the generalizability of the findings to older adults or individuals with different skin characteristics, as these factors can influence PPG-based measurements.
  • Activity Duration: The duration of individual activities was relatively short, which might not be sufficient for optimal HR adaptation in all participants. Longer activity durations could yield more meaningful data.
  • Laboratory Setting: The study was conducted in a laboratory, which may not perfectly replicate real-world usage patterns and environmental factors (e.g., sweat, temperature fluctuations) that can affect device accuracy.
  • Participant 20 Anomaly: One participant exhibited highly erratic heart rate readings from the Samsung Galaxy Watch Active2, which, while included in the main analysis, significantly impacted some metrics. The authors acknowledge the controversy of including this data but opted against exclusion to avoid bias.
  • Future Work: The authors suggest that researchers should carefully consider their specific study requirements when choosing wearables, as factors beyond just HR accuracy, such as raw data access, battery life, and other sensor capabilities, may be equally relevant. The study serves as a prequel to a larger interdisciplinary study in obstetrics, indicating potential for further research in specific populations.

Summary updated

Abstract

<sec> <title>BACKGROUND</title> Fitness trackers and smart watches are frequently used to collect data in longitudinal medical studies. They allow continuous recording in real-life settings, potentially revealing previously uncaptured variabilities of biophysiological parameters and diseases. Adequate device accuracy is a prerequisite for meaningful research. </sec> <sec> <title>OBJECTIVE</title> This study aims to assess the heart rate recording accuracy in two previously unvalidated devices: Fitbit Charge 4 and Samsung Galaxy Watch Active2. </sec> <sec> <title>METHODS</title> Participants performed a study protocol comprising 5 resting and sedentary, 2 low-intensity, and 3 high-intensity exercise phases, lasting an average of 19 minutes 27 seconds. Participants wore two wearables simultaneously during all activities: Fitbit Charge 4 and Samsung Galaxy Watch Active2. Reference heart rate data were recorded using a medically certified Holter electrocardiogram. The data of the reference and evaluated devices were synchronized and compared at 1-second intervals. The mean, mean absolute error, mean absolute percentage error, Lin concordance correlation coefficient, Pearson correlation coefficient, and Bland-Altman plots were analyzed. </sec> <sec> <title>RESULTS</title> A total of 23 healthy adults (mean age 24.2, SD 4.6 years) participated in our study. Overall, and across all activities, the Fitbit Charge 4 slightly underestimated the heart rate, whereas the Samsung Galaxy Watch Active2 overestimated it (−1.66 beats per minute [bpm]/3.84 bpm). The Fitbit Charge 4 achieved a lower mean absolute error during resting and sedentary activities (seated rest: 7.8 vs 9.4; typing: 8.1 vs 11.6; laying down [left]: 7.2 vs 9.4; laying down [back]: 6.0 vs 8.6; and walking slowly: 6.8 vs 7.7 bpm), whereas the Samsung Galaxy Watch Active2 performed better during and after low- and high-intensity activities (standing up: 12.3 vs 9.0; walking fast: 6.1 vs 5.8; stairs: 8.8 vs 6.9; squats: 15.7 vs 6.1; resting: 9.6 vs 5.6 bpm). </sec> <sec> <title>CONCLUSIONS</title> Device accuracy varied with activity. Overall, both devices achieved a mean absolute percentage error of just &amp;lt;10%. Thus, they were considered to produce valid results based on the limits established by previous work in the field. Neither device reached sufficient accuracy during seated rest or keyboard typing. Thus, both devices may be eligible for use in respective studies; however, researchers should consider their individual study requirements. </sec>
BibTeX
@misc{nissen2021heart,
  author = {Michael Nissen and Syrine Slim and Katharina Jäger and Madeleine Flaucher and Hanna Huebner and Nina Danzberger and Peter A. Fasching and Matthias W. Beckmann and Stefan Gradl and Bjoern M. Eskofier},
  title = {{Heart Rate Measurement Accuracy of Fitbit Charge 4 and Samsung Galaxy Watch Active2: Device Evaluation Study (Preprint)}},
  year = {2021},
  doi = {10.2196/preprints.33635},
}
Figure