Fusing Object Information and Inertial Data for Activity Recognition

Alexander Diete, Heiner StuckenschmidtSensors 2019 · OpenReview

TL;DR

This paper introduces a novel multimodal approach to recognize activities of daily living (ADLs) by fusing egocentric video data with inertial sensor data from wearables. By combining object detection with arm movement analysis, the method aims to overcome the limitations of each sensor type, achieving up to 79.6% F₁-measure on a custom ADL dataset.

What Problem Does This Paper Address?

Recognizing activities of daily living (ADLs) is crucial for applications like elder care and health monitoring. While wearable sensors (like inertial measurement units and interaction sensors) are common, they often struggle to differentiate between mere object interaction (e.g., touching) and actual activity completion (e.g., taking medicine). Vision-based approaches, particularly those using egocentric cameras, can identify objects but are limited by field-of-view and can be ambiguous when motions are similar. This paper addresses the challenge of accurately recognizing ADLs by combining the strengths of both egocentric vision (for object context) and inertial sensing (for arm movement dynamics), particularly for activities that are difficult to distinguish based on motion alone.

Methods

The core of the proposed method is a multimodal fusion approach that combines features extracted from egocentric video and inertial sensor data.

  • Vision Features: Object detection is performed using a pre-trained ResNet FPN neural network. The key insight here is to leverage the overlap between detected hands and other objects in the frame as a proxy for interaction. This interaction information is then windowed to create feature vectors.
  • Inertial Features: Acceleration data from smartwatches is used. A sliding window approach is employed, with features extracted in both the time (e.g., mean, median, standard deviation) and frequency domains.
  • Fusion: Both early fusion (concatenating feature vectors) and late fusion (training separate models and combining their outputs) are explored. The paper emphasizes aligning data from different sensors based on temporal overlap and using windowing techniques to create a consistent temporal mapping.

Datasets

The authors utilize two datasets:

  • ADL Dataset (Custom): This dataset was collected by the authors and focuses on activities of daily living that are difficult to distinguish due to similar motions (e.g., eating vs. taking medicine). It includes egocentric video from smart-glasses and chest-mounted tablets, along with inertial data from smartwatches and smartphones.
  • CMU-MMAC Dataset: A publicly available dataset containing recordings of people cooking. The authors use a subset of this dataset, focusing on the "brownie" recipe, and also leverage a recently released set of annotations for a larger subset of the CMU-MMAC dataset.

Key Insights and Ideas

  • Interaction as a Key Indicator: The paper highlights that simply detecting an object is not enough; understanding the interaction between a person's hand and objects is a strong indicator of activity. The overlap of bounding boxes serves as a clever proxy for this interaction.
  • Complementary Strengths: Inertial data excels at capturing arm movement dynamics, while vision data provides crucial object context. Combining them effectively bridges the gap where one modality alone is insufficient (e.g., distinguishing similar motions like eating vs. taking medicine).
  • Windowing for Temporal Alignment: Using temporal windows with defined start and end times, rather than fixed numbers of data points, is crucial for aligning data from sensors with different sampling rates and for robustly handling potential sensor dropouts.
  • Object Detection Limitations: The performance of vision-based recognition is significantly impacted by the quality of object detection. Pre-trained models, while useful, may not perform optimally in diverse or low-quality camera scenarios, leading to a performance gap compared to perfect vision.

Key Results

  • ADL Dataset Performance: The proposed multimodal approach achieved an F₁-measure of up to 79.6% on their custom ADL dataset.
  • CMU-MMAC Dataset Performance: The method achieved an F₁-measure of 58.4% on the original CMU-MMAC dataset and 66.4% on a larger subset with new annotations.
  • Outperforming Baselines: The approach demonstrated superior performance compared to a state-of-the-art method on the CMU-MMAC dataset.
  • Modality Contributions: The paper shows that combining both modalities generally yields better results than using either vision or inertial data alone. Logistic Regression was found to be a strong fusion classifier for the ADL dataset, while Random Forest performed better on the CMU-MMAC dataset.

Limitations and Future Work

  • Object Detection Accuracy: The performance of the vision component is directly tied to the accuracy of the object detection network. In challenging environments or with lower-quality cameras, this can limit overall recognition accuracy. The authors suggest training custom models for specific scenarios to improve this.
  • Privacy Concerns: The use of cameras raises privacy issues. The authors propose solutions like offline processing and minimal object detection to mitigate these concerns.
  • Interaction Ambiguity: While object overlap is a good indicator, it can sometimes be ambiguous, especially in cluttered scenes or when objects are merely passed by.
  • Future Work:
    • Exploring more advanced object detection networks and transfer learning.
  • Investigating a wider range of modalities (e.g., gyration, magnetic field data).
  • Developing more sophisticated fusion techniques, potentially involving different window lengths or voting mechanisms.
  • Improving interaction detection by incorporating motion tracking of objects.
  • Addressing the challenge of egomotion in video data for more accurate interaction analysis.

Summary updated

Abstract

In the field of pervasive computing, wearable devices have been widely used for recognizing human activities. One important area in this research is the recognition of activities of daily living where especially inertial sensors and interaction sensors (like RFID tags with scanners) are popular choices as data sources. Using interaction sensors, however, has one drawback: they may not differentiate between proper interaction and simple touching of an object. A positive signal from an interaction sensor is not necessarily caused by a performed activity e.g., when an object is only touched but no interaction occurred afterwards. There are, however, many scenarios like medicine intake that rely heavily on correctly recognized activities. In our work, we aim to address this limitation and present a multimodal egocentric-based activity recognition approach. Our solution relies on object detection that recognizes activity-critical objects in a frame. As it is infeasible to always expect a high quality camera view, we enrich the vision features with inertial sensor data that monitors the users’ arm movement. This way we try to overcome the drawbacks of each respective sensor. We present our results of combining inertial and video features to recognize human activities on different types of scenarios where we achieve an F 1 -measure of up to 79.6%.
BibTeX
@inproceedings{diete2019fusing,
  author = {Alexander Diete and Heiner Stuckenschmidt},
  title = {{Fusing Object Information and Inertial Data for Activity Recognition}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=Z3-4Fez-8jX},
}
Figure