A multidevice and multimodal dataset for human energy expenditure estimation using wearable devices

Shkurta Gashi, Chulhong Min, Alessandro Montanari, Silvia Santini, Fahim KawsarScientific 2022 · DOI

TL;DR

This paper introduces the WEEE dataset, a rich, multimodal collection of sensor data from multiple wearable devices, alongside ground-truth respiratory measurements, to advance human energy expenditure (EE) estimation. By providing diverse sensor data from various body locations and detailed activity/demographic information, WEEE aims to overcome current limitations in developing robust and accurate EE estimation techniques for real-world applications.

What Problem Does This Paper Address?

Accurately estimating human energy expenditure (EE) is crucial for promoting healthier lifestyles, enabling personalized fitness recommendations, and supporting weight management. While gold-standard methods like direct and indirect calorimetry exist, they are cumbersome, expensive, and impractical for continuous, real-world monitoring. This has led to a surge in research using wearable devices for EE estimation. However, a significant challenge remains: it's unclear which combination of sensors, body locations, and data modalities yields the most reliable EE estimates. Existing datasets often lack diversity in sensor types, body placements, or multimodal data, hindering the development and validation of novel EE estimation techniques. The WEEE dataset aims to fill this gap by providing a comprehensive, multi-device, and multimodal dataset to address these challenges.

Methods

The WEEE dataset was collected through a controlled experiment involving 17 participants performing three distinct activities: resting, cycling, and running, each at two intensity levels. This protocol was designed to capture a range of movement intensities and body movements.

The core of the data collection involved a sophisticated setup of seven wearable devices placed on four body locations: head, ear, chest, and wrist. These devices captured a wide array of physiological signals, including:

  • Respiratory Data: Oxygen consumption (VO2), fraction of oxygen in expired breath (FeO2), and ventilation (Ve) were measured using an indirect calorimeter, serving as the ground truth for EE.
  • Motion Data: Accelerometer (ACC) and gyroscope (GYRO) sensors were used to capture movement patterns.
  • Physiological Signals: Heart rate (HR), heart rate variability (HRV), electrocardiography (ECG), photoplethysmography (PPG), electrodermal activity (EDA), and skin temperature (TEMP) were recorded.
  • Brain Activity: Electroencephalography (EEG) data was also collected from a headband.

In addition to sensor data, the study collected demographic information, body composition data, detailed activity logs (including MET values), and responses to questionnaires covering sleep quality, stress levels, and physical activity habits. This rich, multimodal approach allows for the investigation of how different sensor combinations and physiological signals contribute to EE estimation.

Datasets

The paper introduces the WEEE dataset, which is publicly available for download. This dataset is unique in its:

  • Multidevice and Multimodal Nature: It combines data from seven different wearable devices, capturing a wide range of physiological and motion signals.
  • Diverse Sensor Placement: Data is collected from the head, ear, chest, and wrist, allowing for the exploration of sensor location impact on EE estimation.
  • Ground Truth Data: It includes direct measurements from an indirect calorimeter, providing a reliable benchmark for EE.
  • Comprehensive Metadata: Detailed information on participant demographics, body composition, activity types and intensities (with MET values), and questionnaire responses is provided.

The paper also includes a comparative table (Table 1) of existing datasets for energy expenditure modeling, highlighting how WEEE surpasses many in terms of the number of unique data sources and the inclusion of ACC and HR from multiple body locations.

Key Insights and Ideas

  • The Ear as a Novel Sensor Location: The inclusion of sensors on the ear is a significant contribution, opening up new avenues for EE estimation research. This location is less explored than traditional wrist or chest placements, and its potential for capturing valuable physiological data is highlighted.
  • Multimodality is Key: The dataset's strength lies in its multimodal nature. By combining data from various sensors and body locations, researchers can explore complex fusion strategies to improve EE estimation accuracy, moving beyond single-sensor approaches.
  • Ground Truth is Essential for Validation: The inclusion of indirect calorimetry data is critical. It allows for direct comparison and validation of EE estimates derived from wearable sensors, a crucial step in building trust and reliability in these technologies.
  • Individual Factors Matter: The collection of detailed demographic, body composition, and lifestyle data (sleep, stress, diet) underscores the importance of these factors in influencing EE. This allows for more personalized and context-aware EE estimation models.

Key Results

The paper primarily focuses on presenting and describing the WEEE dataset. While it doesn't present a fully developed EE estimation model, it provides crucial validation of the collected data:

  • Technical Validation: Descriptive statistics of sensor data (Table 5) show that the collected values fall within the expected ranges for most sensors, confirming their validity. For example, HR values from ZEPHYR were within the [25:240] bpm range.
  • Correlation Analysis: A heatmap (Fig. 3) demonstrates significant positive correlations between motion data (ACC, GYRO) collected from different body locations, indicating consistency and reliability.
  • Activity-Based Differences: Visualizations (Fig. 4) show expected patterns in EE, HR, and GYRO data across different activities (resting, cycling, running), with higher intensity activities generally leading to higher values. This confirms that the collected data reflects expected physiological responses to physical exertion.

Limitations and Future Work

The authors acknowledge several limitations and suggest avenues for future research:

  • Limited Number of Physical Activities: The study included only three main activities (resting, cycling, running). Future work should expand this to include a wider variety of physical activities to enhance the dataset's diversity.
  • Potential for Motion Artifacts: The paper notes that low HR values from PPG sensors might be affected by motion artifacts, recommending careful artifact identification and removal.
  • Data Preprocessing Recommendations: The authors provide specific guidance on cleaning and preprocessing data from different devices (e.g., earbuds, wristbands), suggesting the use of libraries like NeuroKit and cvxEDA.
  • Exploration of Sensor Fusion and Location: The dataset is designed to enable research into optimal sensor fusion strategies and the impact of sensor location on EE estimation accuracy.
  • Handling Missing Data: The paper details instances of missing data (e.g., from the MUSE S device and the indirect calorimeter) and highlights the need for methods to handle such issues in future analyses.

Summary updated

Abstract

We present a multi-device and multi-modal dataset, called WEEE, collected from 17 participants while they were performing different physical activities. WEEE contains: (1) sensor data collected using seven wearable devices placed on four body locations (head, ear, chest, and wrist); (2) respiratory data collected with an indirect calorimeter serving as ground-truth information; (3) demographics and body composition data (e.g., fat percentage); (4) intensity level and type of physical activities, along with their corresponding metabolic equivalent of task (MET) values; and (5) answers to questionnaires about participants' physical activity level, diet, stress and sleep. Thanks to the diversity of sensors and body locations, we believe that the dataset will enable the development of novel human energy expenditure (EE) estimation techniques for a diverse set of application scenarios. EE refers to the amount of energy an individual uses to maintain body functions and as a result of physical activity. A reliable estimate of people's EE thus enables computing systems to make inferences about users' physical activity and help them promoting a healthier lifestyle.
BibTeX
@inproceedings{gashi2022multidevice,
  author = {Shkurta Gashi and Chulhong Min and Alessandro Montanari and Silvia Santini and Fahim Kawsar},
  title = {{A multidevice and multimodal dataset for human energy expenditure estimation using wearable devices}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41597-022-01643-5},
}
Figure