Guidelines for wrist-worn consumer wearable assessment of heart rate in biobehavioral research

Benjamin W. Nelson, Carissa A. Low, Nicholas Jacobson, Patricia Areán, John B. Torous, Nicholas B. AllenNpj Digit. Medicine 2020 · DOI

TL;DR

Consumer wearables offer unprecedented opportunities for continuous, real-world heart rate (HR) monitoring in biobehavioral research. However, a lack of standardization in data collection, processing, and reporting hinders reproducibility and generalizability. This paper provides crucial guidelines and checklists to address these challenges, aiming to improve the rigor and impact of research using wearable HR data.

What Problem Does This Paper Address?

The increasing popularity of wrist-worn wearables (smartwatches, fitness trackers) for measuring cardiovascular psychophysiological processes presents a significant opportunity for "big data" research. These devices allow for continuous, scalable, and unobtrusive data collection in real-world settings, overcoming the limitations of traditional laboratory-based measurements. However, this burgeoning field faces a critical challenge: the lack of standardization. This includes inconsistent reporting of device specifics (type, firmware, sampling rate), biobehavioral variables (e.g., body mass index, wrist circumference), and participant demographics (e.g., skin tone). This inconsistency introduces noise, clouds interpretation, and jeopardizes the replicability and generalizability of research findings. The paper aims to establish much-needed guidelines to mitigate these issues.

Methods

The paper doesn't introduce novel algorithms or datasets but rather focuses on establishing a framework for rigorous research practices. The core "method" is the development of comprehensive guidelines and two reporting checklists. These checklists are designed to standardize the reporting of:

  • Study Design and Protocol: Covering aspects from naturalistic vs. laboratory settings to specific inclusion/exclusion criteria.
  • Technological Factors: Detailing device manufacturer, type, version, firmware, sensor type, and sampling rate. Crucially, it emphasizes reporting device reliability and justifying the chosen threshold for the study's design.
  • Participant Characteristics: Mandating the collection and reporting of age, race, ethnicity, biological sex, gender, skin tone (e.g., Fitzpatrick scale), body mass index, wrist circumference, wrist placement, and medical conditions.
  • Data Cleaning, Handling, and Analysis: Outlining the need to report how summary HR metrics are calculated, how non-wear time is defined, how missing data is handled (strongly recommending against listwise deletion), and how outliers are identified and corrected.
  • Data Reporting and Transparency: Advocating for preregistration of analyses, open code and data sharing where applicable, and transparent reporting of descriptive statistics.

The guidelines also discuss potential covariates that can influence HR accuracy, such as technological factors (device differences, firmware updates) and biobehavioral/demographic factors (BMI, skin tone, motion artifacts).

Datasets

Not discussed in the paper. The paper focuses on providing guidelines for future research rather than analyzing specific datasets.

Key Insights and Ideas

  • The "Big Data" Promise and Peril: Wearables offer immense potential for large-scale, ecologically valid data collection, but without standardization, this "big data" can become "noisy data," hindering scientific progress.
  • Beyond Raw Accuracy: While device accuracy is important, the paper highlights that for many research applications, a "better-than-noise" standard might be sufficient, especially when HR is one of many features in a machine learning model.
  • The Importance of "Metadata": The core contribution is the emphasis on comprehensive "metadata" – detailed information about the study design, devices, participants, and data processing. This metadata is crucial for understanding, replicating, and generalizing findings.
  • Demographic and Biobehavioral Factors Matter: Skin tone, BMI, wrist circumference, and even wrist placement can significantly impact HR measurement accuracy. Ignoring these factors can lead to biased results and limit generalizability, particularly for underrepresented populations.
  • Transparency is Key: The paper strongly advocates for open science practices like preregistration and data sharing to combat issues like publication bias and p-hacking, which are amplified in the context of large wearable datasets.
  • Firmware Updates are a Reproducibility Minefield: Differences in firmware versions, even for the same device model, can lead to different results, making it essential to report and potentially control for this factor.

Key Results

The paper does not present quantitative results from a specific study. Instead, its "results" are the proposed guidelines and checklists, which are presented as a solution to the identified problems. The authors synthesize existing literature to demonstrate:

  • Promising Accuracy: Current research suggests that many consumer wearables (e.g., Apple Watch, Fitbit) offer acceptable accuracy for HR measurement in research settings, particularly for resting HR and when data is aggregated and cleaned.
  • Areas of Inaccuracy: Devices can underestimate HR, and accuracy can vary significantly based on activity level, motion artifacts, and individual characteristics.
  • Need for Standardization: The lack of standardized reporting is identified as a major barrier to reproducibility and generalizability, leading to contradictory findings and difficulty in comparing studies.

Limitations and Future Work

Limitations:

  • Not a Replacement for Medical Devices (Yet): The authors acknowledge that while consumer wearables are improving, they are not yet advanced enough to replace medical-grade ECG devices for critical cardiac event monitoring.
  • Focus on HR: The paper primarily focuses on heart rate, acknowledging that other physiological measures (e.g., HRV, blood pressure, oxygen saturation) are also collected by wearables but are outside the scope of this review.
  • Device-Specific Nuances: While guidelines are provided, the rapid pace of technological advancement means that specific device recommendations may quickly become outdated.

Future Work:

  • Establishment of "Metadata Standards": The authors call for the broader adoption of their proposed checklists and the establishment of formal "metadata standards" within the research community.
  • Meta-Analyses on Wearable Accuracy: The need for meta-analyses to summarize the current state of consumer wearable HR accuracy is highlighted.
  • Further Research on Influencing Factors: Continued investigation into how technological, biobehavioral, and demographic factors influence wearable HR accuracy is crucial.
  • Guidelines for Other Physiological Measures: The principles outlined in this paper will likely need to be adapted for other physiological data collected by wearables.
  • Addressing the "Reproducibility Crisis": The paper contributes to the broader effort to combat the reproducibility crisis in science by providing practical tools for improving data transparency and reporting.

Summary updated

BibTeX
@article{nelson2020guidelines,
  author = {Benjamin W. Nelson and Carissa A. Low and Nicholas Jacobson and Patricia Areán and John B. Torous and Nicholas B. Allen},
  title = {{Guidelines for wrist-worn consumer wearable assessment of heart rate in biobehavioral research}},
  journal = {Npj Digit. Medicine},
  year = {2020},
  doi = {10.1038/s41746-020-0297-4},
}
Figure