Evaluation of the Accuracy of Contactless Consumer Sleep-Tracking Devices Application in Human Experiment: A Systematic Review and Meta-Analysis

yonghong yan, Huifang Zhai, Yonghong Yan 0003, Siqi He, Pinyong Zhao, Bohan ZhangSensors 2023 · DOI

TL;DR

Contactless consumer sleep-tracking devices (CCSTDs) show promise as a viable alternative to traditional sleep monitoring methods like polysomnography (PSG) and actigraphy, particularly for large-scale, long-term studies in healthy individuals. While they offer comparable accuracy to actigraphy for sleep/wake detection and can provide valuable sleep stage data, their accuracy can vary, especially in patient populations.

What Problem Does This Paper Address?

Accurate sleep monitoring is crucial for understanding human health and performance, influencing factors like fatigue, mood, and circadian rhythms. Traditional methods like polysomnography (PSG) are considered the gold standard but are expensive, require specialist supervision, and can be uncomfortable, limiting their use in large-scale or long-term studies, especially outside of a lab setting. Actigraphy, while more convenient, is also costly for large cohorts and can be uncomfortable or forgotten by users. This paper addresses the need for more accessible and unobtrusive sleep monitoring solutions by evaluating the accuracy and effectiveness of contactless consumer sleep-tracking devices (CCSTDs) in human experiments.

Methods

The researchers conducted a systematic review and meta-analysis following the PRISMA guidelines. They searched major databases (PubMed, EMBASE, Cochrane CENTRALE, Web of Science) for studies comparing CCSTDs with PSG. Eligibility criteria included commercially available consumer-grade devices, reporting of specific sleep parameters (TST, SOL, WASO, SE, sleep stages, accuracy, sensitivity, specificity), use of PSG or equivalent as a reference, and human participants. Studies with fewer than five subjects, conference papers, or those lacking appropriate statistical analysis were excluded. Data extraction involved collating characteristics of studies, participants, and devices, along with sleep outcomes. Statistical analysis included random-effects models for meta-analysis, heterogeneity assessment using I² statistics, and subgroup analyses to explore the impact of sensor type, device type, participant type, and brand.

Datasets

Not discussed in the paper. The paper focuses on a systematic review and meta-analysis of existing published studies, rather than introducing new datasets.

Key Insights and Ideas

  • CCSTDs as a viable alternative: The study strongly suggests that CCSTDs, particularly mattress-based devices with piezoelectric sensors, can serve as a practical alternative to PSG and actigraphy for large-scale, long-term sleep studies, especially when precise data isn't paramount.
  • Sleep stage data advantage: A significant advantage of CCSTDs over actigraphy is their ability to provide data on sleep stages, which actigraphy typically cannot.
  • Sensor technology matters: The type of sensor plays a crucial role in CCSTD accuracy. Piezoelectric sensors in mattress-based devices showed strong performance, while radiofrequency sensors in bedside devices were also effective, though sometimes with slightly different performance profiles.
  • Healthy vs. Patient populations: CCSTDs demonstrate higher accuracy and reliability in healthy individuals with normal sleep patterns compared to patients with sleep disorders. This is likely due to the increased complexity and variability of sleep in patient populations.
  • Brand variation exists: While some brands like ResMed S+ and Beddit showed good performance, the study acknowledges that there can be significant differences in accuracy and reliability among various CCSTD brands, highlighting the need for brand-specific validation.

Key Results

  • Overall Sleep Parameters: CCSTDs tended to overestimate Total Sleep Time (TST) and Sleep Efficiency (SE), and underestimate Sleep Onset Latency (SOL) and Wake After Sleep Onset (WASO) when compared to PSG. Deep sleep was also overestimated, while light sleep and REM showed non-significant differences.
  • Epoch-by-Epoch (EBE) Agreement: For sleep/wake detection, CCSTDs showed high sensitivity (0.90 ± 0.06) but lower specificity (0.51 ± 0.12), indicating a tendency to correctly identify sleep but struggle with accurately identifying wakefulness. Accuracy ranged from 0.68 to 0.91.
  • Sleep Stage Identification: CCSTDs showed a mixed performance in sleep stage identification, with relatively low sensitivity for light sleep, deep sleep, and REM, but higher specificity for deep sleep and REM. Overall accuracy for sleep stages was moderate.
  • Comparison with Actigraphy: CCSTDs demonstrated similar accuracy to actigraphy for sleep/wake detection, but generally had lower sensitivity and higher specificity. TST, SE, SOL, and WASO estimations were often superior with CCSTDs compared to actigraphy.

Limitations and Future Work

The authors acknowledge several limitations:

  • Short monitoring duration: Most included studies monitored participants for only one night, limiting the assessment of long-term reliability.
  • Lack of detailed information: There's a scarcity of published information regarding the specific algorithms and hardware reliability of various CCSTDs, making it difficult to fully explain performance differences.
  • Limited sleep indicators: Some studies focused on only one or two sleep indicators, potentially introducing bias.
  • Market diversity: The wide variety of CCSTDs available, with varying levels of standardization and validation, poses a challenge.

Future work should focus on:

  • Long-term validation: Studies with longer monitoring periods are needed to assess the sustained accuracy and reliability of CCSTDs.
  • Algorithm and hardware transparency: More detailed information on the underlying technology of CCSTDs would facilitate better understanding and comparison.
  • Population-specific validation: Further validation studies are recommended for specific patient populations to understand their unique sleep characteristics and how CCSTDs perform in these contexts.
  • Standardized testing frameworks: Developing standardized protocols for testing CCSTDs is crucial for consistent and reliable evaluations.

Summary updated

BibTeX
@article{yan2023evaluation,
  author = {yonghong yan and Huifang Zhai and Yonghong Yan and Siqi He and Pinyong Zhao and Bohan Zhang},
  title = {{Evaluation of the Accuracy of Contactless Consumer Sleep-Tracking Devices Application in Human Experiment: A Systematic Review and Meta-Analysis}},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23104842},
}
Figure