Exploring Semi-Supervised Methods for Labeling Support in Multimodal Datasets

Alexander Diete, Timo Sztyler, Heiner StuckenschmidtSensors 2018 · OpenReview

TL;DR

This paper introduces a semi-supervised approach to accelerate the annotation of multimodal activity recognition datasets, specifically leveraging wrist-worn accelerometer data to suggest labels for video recordings. By learning from a small set of manually labeled examples, their system can then automatically propose labels for the rest of the dataset, significantly reducing the annotation burden.

What Problem Does This Paper Address?

Annotating multimodal datasets, especially those involving video and sensor data, is a major bottleneck in activity recognition research. Watching entire video recordings to identify and label specific activities is incredibly time-consuming. While existing tools might offer visualization aids, they often lack automated suggestions. This paper tackles the challenge of making the annotation process more efficient by developing a system that can infer and recommend labels based on a limited initial annotation effort, thereby reducing the manual workload for researchers and domain experts.

Methods

The core of the proposed method relies on template matching using Dynamic Time Warping (DTW). The process involves:

  1. Data Alignment: Synchronizing data from different sensors (e.g., smartglasses and wristbands) and aligning them with video recordings, often using distinct motion patterns or "zerolines" as reference points.
  2. Template Creation: Manually labeling a small subset of the data to create "templates" representing specific activities. These templates capture the temporal sequence of sensor data (primarily acceleration) associated with an activity.
  3. Semi-Supervised Labeling: Applying DTW to match these learned templates against the unlabeled sensor data. DTW is crucial here as it can find similarities between time series of different lengths, accounting for variations in activity execution speed.
  4. Candidate Selection: Developing strategies to select the most likely matches from the DTW results. This includes methods like using "zeroline" templates to filter out irrelevant matches and ensuring selected candidates are temporally distinct.
  5. Label Recommendation: Presenting the identified matches as suggested labels to the annotator, who can then confirm, correct, or reject them.

The authors also explore different preprocessing techniques for the acceleration data, such as using linear acceleration versus raw data, and reducing the three-dimensional data to a single dimension, to optimize matching performance.

Datasets

The authors evaluated their approach on three distinct datasets:

  1. Picking Dataset: A custom dataset focusing on warehouse picking activities, involving grabbing items from shelves. This dataset was collected using smartglasses and a custom wristband.
  2. Activities of Daily Living (ADL) Dataset: Another custom dataset featuring activities like eating, drinking water, and taking medicine, performed while lying down. This dataset was collected using smartwatches and smartphones.
  3. CMU-MMAC Dataset: A publicly available, more complex dataset containing kitchen activities, used as a reference.

Key Insights and Ideas

  • Leveraging Inertial Data for Video Annotation: The paper demonstrates that accelerometer data from a simple wrist-worn device can be surprisingly effective in identifying and suggesting labels for video recordings, even when the activities are complex.
  • Dynamic Time Warping's Suitability: DTW proves to be a robust technique for matching activity templates, effectively handling variations in the speed and duration of human movements across different individuals and recordings.
  • The Importance of Granularity: The effectiveness of the approach is highly dependent on the granularity of the activity labels. Simple, distinct motions are easier to template and match than complex activities that involve multiple sub-movements.
  • Clustering for Robust Templates: For more complex activities, clustering extracted templates based on their DTW distances can reveal underlying motion similarities, leading to more generalized and robust templates.
  • Hand Dominance Matters: In multimodal datasets, differences in performance between left and right hand data can arise, potentially due to natural hand dominance, impacting the consistency of recognition.

Key Results

  • Picking Dataset: The method achieved a high overlap (average 69%) for detecting grabbing motions, though accuracy in precise start/end boundaries was a challenge.
  • ADL Dataset: The best performing configuration involved using linear acceleration, reducing dimensions to one, and employing a delta-based candidate selection method, achieving a median error of 0.8s and 0.5s for subjects.
  • Recall and ROC Curves: For the ADL dataset, recall values up to 0.8 were achieved with appropriate k-values, and ROC curves indicated acceptable performance, particularly for the dominant (right) hand.
  • Template Efficiency: The study suggests that a relatively small number of templates (e.g., two) can be sufficient to achieve reasonable results for certain activities.

Limitations and Future Work

  • Boundary Accuracy: The paper acknowledges that precisely identifying the start and end boundaries of activities remains a challenge, partly due to the fixed length of templates.
  • Activity Granularity: The effectiveness of the approach is sensitive to the granularity of the activity labels. Complex activities might require breaking them down into simpler sub-activities.
  • Sensor Modalities: While focusing on acceleration data, the authors suggest that incorporating other sensor modalities could further improve recognition accuracy.
  • User Study: A crucial next step is a user study to quantitatively measure the impact of the proposed labeling support on annotation time and agreement among annotators.
  • Dataset Enhancement: Expanding the dataset with more instances and a wider range of activities is planned to improve generalization.

Summary updated

Abstract

Working with multimodal datasets is a challenging task as it requires annotations which often are time consuming and difficult to acquire. This includes in particular video recordings which often need to be watched as a whole before they can be labeled. Additionally, other modalities like acceleration data are often recorded alongside a video. For that purpose, we created an annotation tool that enables to annotate datasets of video and inertial sensor data. In contrast to most existing approaches, we focus on semi-supervised labeling support to infer labels for the whole dataset. This means, after labeling a small set of instances our system is able to provide labeling recommendations. We aim to rely on the acceleration data of a wrist-worn sensor to support the labeling of a video recording. For that purpose, we apply template matching to identify time intervals of certain activities. We test our approach on three datasets, one containing warehouse picking activities, one consisting of activities of daily living and one about meal preparations. Our results show that the presented method is able to give hints to annotators about possible label candidates.
BibTeX
@inproceedings{diete2018exploring,
  author = {Alexander Diete and Timo Sztyler and Heiner Stuckenschmidt},
  title = {{Exploring Semi-Supervised Methods for Labeling Support in Multimodal Datasets}},
  booktitle = {Sensors},
  year = {2018},
  url = {https://openreview.net/forum?id=AABlgsAcRAJ},
}
Figure