On Reliability of Android Wearable Health Devices

Naixing Wang, Edgardo Barsallo Yi, Saurabh BagchiarXiv 2017 · arXiv

Introduction

As wearable health monitors transition from niche gadgets to mainstream clinical tools, their reliability remains largely a "black box." Most consumers rely on these devices for fitness tracking, yet they primarily utilize photoplethysmography (PPG)—an optical method of measuring heart rate that is notoriously sensitive to physical movement. This study investigates the reliability of Android wearables, specifically the Motorola Moto 360 and Apple Watch, by benchmarking them against clinical-grade equipment and subjecting their software to stress tests.

Methodology

The researchers employed a multi-faceted experimental approach to evaluate both hardware and software reliability:

  1. Hardware Accuracy Study: The heart rate readings of the Motorola Moto 360 (2nd Gen) and the Apple Watch (1st Gen) were compared against a "golden reference"—a clinical-grade fingertip pulse oximeter. Subjects were tested at rest and while jogging at 5 mph.
  2. Motion Correlation Analysis: To isolate the cause of PPG signal degradation during exercise, the team analyzed movements along the X, Y, and Z axes. They synchronized 3D accelerometer data with PPG sensor readings to identify which specific direction of motion causes the most interference.
  3. Software Fuzz Testing: Using the Android Monkey tool, the researchers tested OS stability by sending pseudo-random streams of user events (clicks, swipes, etc.) to built-in fitness apps. They conducted an initial 4,000 trials per app, followed by 10,000 trials for apps showing instability.

Hardware Performance and Motion Interference

The study revealed that while wearables are highly accurate at rest, physical activity significantly degrades data integrity. At rest, both the Apple Watch and Moto 360 tracked closely with the fingertip oximeter. However, during a 5 mph jog, the discrepancy became pronounced.

Figure 1: Heart rate comparison between the Apple Watch, Moto 360 G2, and a clinical finger clip at rest and during exercise.

As shown in Figure 1, the Apple Watch generally remained closer to the reference values than the Moto 360 during exercise, although both devices exhibited lag and errors. A key finding was the identification of the "X-Axis" Culprit. The researchers found that PPG signal distortion is most highly correlated with acceleration along the X-axis (side-to-side wrist movement), with a correlation coefficient of 0.53.

Figure 2: Analysis of PPG signal (LED1) correlation with X, Y, and Z axis acceleration.

Figure 2 illustrates the relationship between motion artifacts and the PPG signal. The data suggests a non-negligible delay of approximately 0.2 seconds between physical movement and the resulting noise in the PPG signal. This indicates that specific arm swing patterns during running are particularly damaging to heart rate data accuracy.

Software Stability and Fuzz Testing

Beyond hardware sensors, the study examined the stability of the underlying Wear OS applications. Surprisingly, core fitness apps were found to be the least reliable components.

During the fuzz testing phase, the Google Fitness app crashed once, while the Motorola Body app crashed six times. The primary cause for all recorded crashes was a NullPointerException. This is a common programming error where an application attempts to access data that does not exist, suggesting that even flagship fitness software lacks robust error handling under high-stress user interaction.

Conclusion and Future Directions

The research highlights a significant gap between the intended use of wearables for fitness and their actual performance under active conditions. While the Apple Watch outperformed the Moto 360 in accuracy, both are susceptible to motion artifacts, particularly those originating from X-axis acceleration.

Future work will move toward White-Box Testing to analyze the source code of Wear OS more deeply. Additionally, the researchers plan to implement Sensor Fuzzing, sending corrupted or extreme sensor data to the OS to test its resilience. Ultimately, these findings could lead to improved noise cancellation algorithms that leverage X-axis correlation to filter out motion artifacts in real-time.

Summary updated

Abstract

Wearable devices are increasingly being used for monitoring health signals and for fitness purposes with typical uses being calorie tracker, workout assistant, and step counter. Even though these wearables can measure many health signals (e.g. heart rate), they are still not perceived as highly accurate, relative to clinical monitoring devices. In this paper, we investigate the accuracy of heart monitor as included in two popular wearables Motorola Moto 360 and the Apple Watch. We analyze the accuracy from a hardware and a software perspective and show the effects of body motion on the heart rate monitors based on the use of photoplethysmography (PPG) signals used in Android wearables. We then do a software reliability study of the Android Wear OS, on which many wearables are based, using fuzz testing.
BibTeX
@misc{wang2017reliability,
  author = {Naixing Wang and Edgardo Barsallo Yi and Saurabh Bagchi},
  title = {{On Reliability of Android Wearable Health Devices}},
  year = {2017},
  eprint = {1706.09247},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.09247},
}
Figure