TL;DR
This paper introduces SensorHub, a novel system designed to simplify the collection of multimodal sensor data from various wearable devices in real-life, home-based study settings. By providing a unified platform for data aggregation and enabling ecological momentary assessments (EMAs), SensorHub aims to bridge the gap between controlled lab experiments and real-world observations, facilitating more ecologically valid research.
What Problem Does This Paper Address?
Conducting observational studies in real-world environments is crucial for understanding human behavior and validating findings from controlled experiments. However, collecting data from multiple wearable sensors, especially those from different manufacturers, presents significant technical challenges. These include managing diverse data formats, synchronizing data streams, and the sheer effort required from both participants and researchers. Existing solutions often fall short by focusing on single-device types, relying on proprietary data formats, or lacking the flexibility for home-based studies. This paper addresses the need for a robust, user-friendly, and interoperable system that can seamlessly integrate data from various wearable sensors and incorporate subjective user feedback through EMAs, all within a home-based setting.
Methods
SensorHub is built around a modular architecture comprising three main components: a smartphone app (Android), a backend server, and a researcher dashboard.
- Smartphone App: This is the primary interface for the study participant. It connects to various wearable devices via Bluetooth (classic and BLE), aggregates their data into a consistent format, and streams it to the backend in real-time. The app also supports the integration of Ecological Momentary Assessments (EMAs) through customizable questionnaires (e.g., Likert scales, affective sliders) and audio recordings.
- Backend: This component acts as the central data repository. It utilizes a combination of PostgreSQL for metadata and study configurations and InfluxDB for efficient storage and retrieval of time-series sensor data. A Ktor web server exposes a REST API for data ingestion and retrieval.
- Dashboard: This web-based interface allows researchers to manage studies, define sensors and their placements, configure EMAs, and visualize the collected data. It's built using Flask for the web framework and Bokeh for interactive plotting.
The system emphasizes modularity, allowing for the easy integration of new sensors by subclassing a Device superclass. Dynamic delivery modules enable the app to load only necessary sensor modules, reducing storage footprint on the user's device.
Datasets
The paper describes two projects that utilized SensorHub for data collection:
- Exercise Exertion Prediction Project: This project collected IMU (6 sensors) and ECG data from 16 participants performing resistance training. Data was streamed to the SensorHub app on an Android smartphone. Subjective exhaustion was measured using the Borg scale.
- EatMaps Project: This project observed how affective states and user contexts correlate with eating habits. Data was collected from 11 subjects over extended periods, including physiological and movement signals from Shimmer3 GSR+ devices (sampling at 50-52 Hz), internal smartphone sensors (IMU, GPS, light, Bluetooth), and EMA data (questionnaires on stress, emotions, wake-up time, social interaction, background noise, and voice prompts). Subjects also took photos of their meals.
No new datasets were introduced; rather, SensorHub was used to collect data for these existing research projects.
Key Insights and Ideas
- Bridging the Lab-Real World Divide: The core idea is that home-based, multimodal sensing is essential for ecological validity, moving beyond the limitations of sanitized laboratory environments.
- Interoperability is Key: The significant challenge of integrating devices from different manufacturers is tackled by SensorHub's design, which aims for a unified data format and communication protocol.
- EMAs Enhance Data Richness: Combining objective sensor data with subjective user feedback (EMAs) provides a more holistic understanding of human behavior and context.
- Modularity for Scalability: The system's modular architecture makes it adaptable to new sensors and future research needs, crucial for a rapidly evolving field.
- Consumer Hardware Focus: Prioritizing consumer-grade wearables makes studies less intrusive and more feasible for participants in their daily lives.
Key Results
- Successful Multimodal Data Collection: SensorHub was successfully used in two pilot studies involving 11 and 16 participants, respectively, demonstrating its capability to collect data from multiple wearable sensors simultaneously.
- Technical Feasibility: Trials with up to 21 participants recording simultaneously showed that data loss is rare, even with high sampling frequencies (e.g., 1000 Hz for ECG), suggesting that theoretical data rate limitations are often not a practical issue.
- Improved Model Performance: In the exercise exertion prediction project, combining IMU and HRV data (collected via SensorHub) significantly improved model performance (higher R² and lower MSE) compared to using IMU data alone.
- Battery Efficiency: Early measurements indicated that using BLE for data transmission with multiple devices for about an hour resulted in less than 3% battery drain on the smartphone.
- Backend Performance: A stress test demonstrated that the SensorHub backend can handle a high volume of incoming data, capable of ingesting data from multiple users attempting to upload simultaneously after a period of caching.
Limitations and Future Work
- Data Rate and Packet Loss: While theoretical limitations exist for BLE data rates and packet loss, practical studies showed these were not significant issues for typical use cases. However, high-demand scenarios like continuous EEG recording with many electrodes might still pose challenges.
- Number of Sensors: Android devices have hardware and software limitations on the number of simultaneous BLE connections (around 7 for older versions, but this can vary by device).
- Missing Raw Data Access: SensorHub currently relies on sensors that provide raw data access. Some consumer wearables only offer pre-processed data, limiting their integration.
- Functionality vs. Simplicity Trade-off: The authors acknowledge the ongoing balance between offering extensive functionality and maintaining a simple, user-friendly interface.
- Future Work:
- Further iterations of user studies to refine the system and improve user-friendliness for both subjects and researchers.
- Implementation of more sensor types to expand the system's versatility.
- Potentially developing a more comprehensive platform approach with enhanced functionality, while still managing complexity.
Summary updated