Automated Quality Control for Sensor Based Symptom Measurement Performed Outside the Lab

Reham Badawy, Yordan P. Raykov, Luc J. W. Evers, Bastiaan R. Bloem, Marjan J. Faber, Andong Zhan, Kasper Claes, Max A. LittleSensors 2018 · DOI

TL;DR

This paper introduces a novel algorithmic framework for automatically assessing the quality of sensor data collected during remote clinimetric tests, crucial for objective symptom measurement outside controlled lab settings. By effectively identifying and filtering out unreliable data, the system achieves high accuracy (around 90%) in segmenting valid measurements, thereby improving the reproducibility and reliability of remote health monitoring.

What Problem Does This Paper Address?

Monitoring patient symptoms using wearable sensors and smartphones offers a promising avenue for objective, non-invasive, and remote health assessment. However, a significant challenge arises when this data is collected outside of controlled laboratory environments. Unpredictable user behaviors and confounding environmental factors can lead to data that violates the assumptions of clinimetric testing protocols, ultimately compromising the accuracy of subsequent analysis and scientific conclusions. Traditional activity recognition methods often struggle with this due to their reliance on hand-crafted features and labeled data, which are rarely available in real-world remote monitoring scenarios. This paper addresses the critical need for an automated system that can reliably distinguish between useful and irrelevant sensor data in these challenging real-world conditions.

Methods

The authors propose a unified algorithmic framework for automated sensor data quality control. The core of their approach involves:

  • Sensor-Specific Preprocessing: Tailored steps to remove confounding factors from raw sensor data. For accelerometry data, this includes isolating and removing the effect of Earth's gravitational field using an L1-trend filtering method. For voice data, it involves segmenting the signal into short windows and extracting energy.
  • Unsupervised Behavior Modeling (Segmentation): Two primary methods are proposed for segmenting the preprocessed data into intervals representing distinct behaviors:
  • Gaussian Mixture Model (GMM): A simpler approach that clusters data into two components, assuming adherence and violation of protocols fall into different Gaussian distributions. A running median filter is used to smooth out rapid switching.
  • Nonparametric Switching Autoregressive (AR) Model: A more flexible and general approach that models the data using a sequence of AR models. This method can accommodate an unknown and changing number of behavioral states, making it suitable for complex scenarios. It leverages an "infinite HMM" concept, allowing the model to adapt to new patterns without pre-specifying the number of states.
  • Context Mapping (Classification): A multinomial naive Bayes classifier is trained to map the segmented behavioral states to adherence or violation of the test protocol. This classifier is trained using the state assignment probabilities from the switching AR model and binary labels (adherence/violation). The authors emphasize the interpretability of this classifier.

Datasets

The primary dataset used for evaluation is from the Smartphone-PD study. This dataset includes:

  • Data from 300 clinimetric tests (100 walking, 100 balance, and 100 voice tests).
  • Tests performed by both Parkinson's disease (PD) patients and healthy controls.
  • Data collected from smartphone sensors (accelerometer, microphone).

The authors also conducted controlled experimental smartphone tests with healthy individuals to generate "ground truth" labels for specific behaviors (walking, balance, voice) to aid in understanding and validating the segmentation process. These controlled tests involved varying phone orientation and location and instructing subjects to perform common behaviors.

Key Insights and Ideas

  • The "Outside the Lab" Problem is Universal: The paper strongly emphasizes that the challenges of data quality control due to unpredictable user behavior and environmental confounders are not unique to PD but are a general problem for remote sensor-based health monitoring.
  • Unsupervised Segmentation is Key: The reliance on unsupervised learning for segmentation is crucial because obtaining detailed, labeled behavioral data in real-world remote monitoring is often infeasible.
  • Flexibility of the Nonparametric Switching AR Model: The ability of the nonparametric switching AR model to adapt to an unknown and changing number of behavioral states is a significant advantage over traditional fixed-state models, making it more robust to the variability of real-world data.
  • Interpretability Matters: The choice of a naive Bayes classifier for context mapping is motivated by its interpretability, which is important for understanding the model's decisions in a healthcare context.
  • Single Feature Sufficiency: The framework demonstrates that high performance can be achieved using a single, carefully chosen feature per sensor type, bypassing the need for extensive, sensor-specific feature engineering.

Key Results

  • The proposed framework achieves an average segmentation accuracy of around 90% when compared to human experts.
  • The nonparametric switching AR model combined with a naive Bayes classifier demonstrated strong performance across different clinimetric tests:
  • Walking Tests: Balanced Accuracy (BA) 85% (11%), True Positive (TP) 85% (18%), True Negative (TN) 90% (8%).
  • Balance Tests: BA 81% (14%), TP 81% (16%), TN 88% (9%).
  • Voice Tests: BA 89% (8%), TP 88% (9%), TN 91% (9%).
  • The GMM-based approach showed high accuracy for voice tests (BA 99%) but significantly lower performance for walking and balance tests, highlighting the limitations of simpler models for complex dynamics.
  • A randomized classifier (trained on shuffled state indicators) achieved balanced accuracy around 50%, confirming that the performance of the proposed methods is due to meaningful segmentation rather than random chance.

Limitations and Future Work

  • Limitations:
    • The authors acknowledge that the largely unsupervised nature of the approach might lead to difficulties in locating clusters of composite human behaviors. For instance, complex activities like cooking involve multiple sub-activities, which might require more labeled data and sophisticated segmentation models.
  • The current framework is primarily demonstrated for simple physical activities.
  • Future Work:
    • Simultaneous Multimodal Sensing: Extending the framework to preprocess and model multiple sensor types simultaneously to potentially increase quality control accuracy.
  • Real-Time Deployment: Incorporating the system into PD research apps (like mPower) for automatic, real-time quality control, eliminating the need to store or analyze unwanted data. This would involve optimizing the computationally intensive switching AR model inference for resource-constrained devices.
  • Contextual Learning for Passive Monitoring: Training the system to recognize specific patient activities from continuous, passive sensor data to assist researchers in objectively testing hypotheses, especially when labeled instances of relevant behaviors are scarce.

Summary updated

BibTeX
@article{badawy2018automated,
  author = {Reham Badawy and Yordan P. Raykov and Luc J. W. Evers and Bastiaan R. Bloem and Marjan J. Faber and Andong Zhan and Kasper Claes and Max A. Little},
  title = {{Automated Quality Control for Sensor Based Symptom Measurement Performed Outside the Lab}},
  journal = {Sensors},
  year = {2018},
  doi = {10.3390/s18041215},
}
Figure