TL;DR
This paper introduces a low-cost, privacy-preserving edge computing system that simultaneously captures diverse ambient data (video, audio, environmental sensors, Bluetooth signals) in clinical and home settings. The system enables real-time analysis of patient behavior and environmental conditions, offering a more holistic approach to monitoring than traditional methods.
What Problem Does This Paper Address?
Current patient monitoring systems, whether in clinical settings like ICUs or for home-based studies, often struggle with data integration, lack spatiotemporal precision for key events, and raise significant privacy concerns. Wearable sensors can be intrusive and suffer from data loss due to motion artifacts, while existing non-contact systems may not capture a comprehensive set of data modalities or are prohibitively expensive. This paper aims to bridge this gap by developing an affordable, privacy-conscious system that captures a rich tapestry of data from the patient's environment and behavior.
Methods
The core of the system is a Raspberry Pi (RPi) acting as an edge computing hub. This allows for decentralized data processing, reducing latency and bandwidth requirements. The system integrates several low-cost sensors:
- PIR Sensor: For basic human movement detection.
- RPi-IR Camera: Captures infrared video for more detailed movement analysis, including intensity and direction. Techniques like Global Difference Sum (GDS) and Global d-Pixel Count (GDPC) are used to extract movement signals.
- USB Microphone: Captures audio for medical equipment alarm classification. Features like Mel-frequency cepstral coefficients (MFCC) and spectral energy are extracted.
- TCS34725 Color Sensor: Measures ambient light intensity (LUX) and color temperature.
- DHT22 Temperature-Humidity Sensor: Records ambient temperature and humidity.
- Bluetooth Beacon: Used for human geolocation within a built environment, tracking signal strength (RSSI).
- Coral USB TPU Accelerator: Enables real-time human pose estimation using PoseNet, extracting keypoint representations that preserve privacy.
Data is processed at the edge, with raw video and audio discarded after feature extraction, and only processed features are sent to a HIPAA-compliant cloud storage for retrospective analysis.
Datasets
- Medical Equipment Alarm Sounds: The authors utilized an external database of medical equipment alarm sounds, specifically those conforming to the ISO/IEC 60601-1-8 standard. This dataset comprised eight categories of alarms, with nine musical notes used to construct 16 different alarm audio recordings.
- Oxford Lip Reading Sentences 2 Dataset: This dataset was used to create mixed audio recordings of alarm sounds and speech to test the robustness of the alarm classification system in noisy environments.
- Human Activity Data: For occupancy estimation and activity phenotyping, a custom three-minute structured video was recorded with three subjects performing standing and hand-exercising actions.
- Ambient Light Data: Two consecutive days of ambient light data were collected in a built environment, along with corresponding weather (cloud cover) and time-of-day information.
- Temperature and Humidity Data: 72 hours of temperature and humidity data were collected and compared against a commercial sensor.
Key Insights and Ideas
- Privacy-Preserving Edge Computing: The system's core innovation lies in its ability to perform significant data processing and feature extraction at the edge, discarding raw, sensitive data (like video and audio) to maintain patient privacy. This is crucial for widespread adoption in healthcare.
- Multimodal Data Fusion: By integrating diverse sensor modalities, the system captures a much richer picture of the patient's state and environment than single-modality systems. This allows for more comprehensive analysis, such as correlating environmental factors with patient behavior.
- Low-Cost and Accessible Hardware: The reliance on readily available and inexpensive hardware like the Raspberry Pi makes the system highly scalable and deployable in various settings, democratizing advanced monitoring capabilities.
- Ambient Environment as a Key Indicator: The paper strongly emphasizes the importance of ambient conditions (light, temperature, humidity) not just for patient comfort but also for their health outcomes and recovery, highlighting their role in circadian rhythms and preventing issues like fungal growth.
Key Results
- Occupancy Estimation: Achieved 94% accuracy in estimating occupancy from video.
- Medical Equipment Alarm Classification: Demonstrated high performance with micro-averaged F1 scores of 0.98 (without speech) and 0.93 (with speech mixing), and macro-averaged F1 scores of 0.97 (without speech) and 0.91 (with speech mixing).
- Human Geolocation: Achieved a room-level accuracy of 98.7% in a built environment.
- Temperature and Humidity Logging: RMSE of 0.28 °C for temperature and 1.00%RH for humidity compared to a commercial sensor, with high correlation (r² = 0.97 for temperature, r² = 0.90 for humidity).
- Ambient Light Logging: Showed a median difference in illuminance of 44.15 LUX between clear and cloudy days.
Limitations and Future Work
- Human Geolocation: The current geolocation method relies on a Bluetooth beacon carried by the subject, which is not entirely non-contact.
- Occupancy Estimation: The current method for occupancy estimation has a drawback with occlusion (when objects merge), though the authors suggest temporal tracking and interpolation as a future solution.
- Future Work: The authors plan to perform multimodal analysis on the larger dataset collected and discover features to detect patients with high-severity neurological diseases. They also aim to address the occlusion issue in occupancy estimation. The system's extensibility allows for incorporating other feature extraction algorithms and sensor modalities.
Summary updated