Reconsideration on evaluation of machine learning models in continuous monitoring using wearables

Cheng Ding, Zhicheng Guo, Cynthia Rudin, Ran Xiao, Fadi B. Nahab, Xiao HuarXiv 2023 · OpenReview

Abstract: This paper explores the challenges in evaluating machine learning (ML) models for continuous health monitoring using wearable devices beyond conventional metrics. We state the complexities posed by real-world variability, disease dynamics, user-specific characteristics, and the prevalence of false notifications, necessitating novel evaluation strategies. Drawing insights from large-scale heart studies, the paper offers a comprehensive guideline for robust ML model evaluation on continuous health monitoring.

Introduction

The rapid proliferation of wearable health technology has led to a surge in machine learning (ML) models designed for continuous physiological monitoring. However, a significant gap exists between the high performance reported in controlled laboratory settings and the actual utility of these devices in the real world. Standard evaluation metrics, such as segment-level accuracy or F1 scores, often fail to account for the noise and variability inherent in daily life.

Challenges in evaluating ML models for continuous health monitoring using wearables

The Challenges of Continuous Monitoring

Evaluating ML models for wearables involves navigating four primary complexities:

  1. Real-world Variability: Signals like photoplethysmography (PPG) are highly sensitive to movement, ambient light, and seasonal changes.
  2. User-specific Characteristics: Biological differences, including skin tone (melanin levels), age, and medical history, significantly impact signal quality and model generalizability.
  3. False Notifications: High-frequency predictions can lead to "alarm fatigue" if not properly mitigated through aggregation strategies.
  4. Dynamic Nature of Diseases: Conditions like Atrial Fibrillation (AF) are intermittent. Segment-level analysis often misses the long-term dynamics and overall disease burden.

A Four-Pillar Evaluation Guideline

To address these challenges, we propose a standardized framework for evaluating wearable ML models:

  • Cohort Selection: Evaluation must include diverse demographics to ensure equitable performance across different skin tones and age groups.
  • Target Scenarios: Models should be tested across various activity levels (e.g., sedentary, active, sleep) rather than just stationary states.
  • Aggregation Strategies: Instead of relying on isolated segments, models should aggregate data over time (e.g., requiring multiple irregular readings within a specific window) to improve clinical relevance.
  • Novel Metrics: We introduce metrics that better reflect real-world robustness.
PPG morphology changes over time, illustrating the dynamic nature of physiological signals

Quantifying Robustness: The Artifact Tolerance Index (ATI)

A key contribution of this work is the Artifact Tolerance Index (ATI). By plotting performance metrics against signal quality—creating an Artifact Tolerance Curve (ATC)—researchers can quantify how "brittle" or "robust" a model is. A high ATI indicates that a model maintains its predictive power even as signal quality degrades due to motion or environmental factors.

Insights from Large-Scale Studies

Reviewing landmark studies such as the Apple, Fitbit, and HUAWEI heart studies reveals critical lessons. For instance, the Apple Heart Study achieved high precision but low sensitivity, highlighting the trade-offs in aggregation strategies. Furthermore, the lack of open-access data from these commercial trials hinders the development of standardized, reproducible evaluation benchmarks.

Conclusion

Moving forward, the evaluation of wearable health tech must shift from statistical segment-level metrics to user-centric, clinically meaningful outcomes. By adopting the ATI and focusing on disease burden and onset/offset estimation, the field can develop more reliable and equitable monitoring solutions.

Summary updated

Abstract

This paper explores the challenges in evaluating machine learning (ML) models for continuous health monitoring using wearable devices beyond conventional metrics. We state the complexities posed by real-world variability, disease dynamics, user-specific characteristics, and the prevalence of false notifications, necessitating novel evaluation strategies. Drawing insights from large-scale heart studies, the paper offers a comprehensive guideline for robust ML model evaluation on continuous health monitoring.
BibTeX
@misc{ding2023reconsideration,
  author = {Cheng Ding and Zhicheng Guo and Cynthia Rudin and Ran Xiao and Fadi B. Nahab and Xiao Hu},
  title = {{Reconsideration on evaluation of machine learning models in continuous monitoring using wearables}},
  year = {2023},
  eprint = {2312.02300},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.02300},
}
Figure