TL;DR
This paper introduces a semi-supervised approach to accelerate the annotation of multimodal activity recognition datasets, specifically leveraging wrist-worn accelerometer data to suggest labels for video recordings. By learning from a small set of manually labeled examples, their system can then automatically propose labels for the rest of the dataset, significantly reducing the annotation burden.
What Problem Does This Paper Address?
Annotating multimodal datasets, especially those involving video and sensor data, is a major bottleneck in activity recognition research. Watching entire video recordings to identify and label specific activities is incredibly time-consuming. While existing tools might offer visualization aids, they often lack automated suggestions. This paper tackles the challenge of making the annotation process more efficient by developing a system that can infer and recommend labels based on a limited initial annotation effort, thereby reducing the manual workload for researchers and domain experts.
Methods
The core of the proposed method relies on template matching using Dynamic Time Warping (DTW). The process involves:
- Data Alignment: Synchronizing data from different sensors (e.g., smartglasses and wristbands) and aligning them with video recordings, often using distinct motion patterns or "zerolines" as reference points.
- Template Creation: Manually labeling a small subset of the data to create "templates" representing specific activities. These templates capture the temporal sequence of sensor data (primarily acceleration) associated with an activity.
- Semi-Supervised Labeling: Applying DTW to match these learned templates against the unlabeled sensor data. DTW is crucial here as it can find similarities between time series of different lengths, accounting for variations in activity execution speed.
- Candidate Selection: Developing strategies to select the most likely matches from the DTW results. This includes methods like using "zeroline" templates to filter out irrelevant matches and ensuring selected candidates are temporally distinct.
- Label Recommendation: Presenting the identified matches as suggested labels to the annotator, who can then confirm, correct, or reject them.
The authors also explore different preprocessing techniques for the acceleration data, such as using linear acceleration versus raw data, and reducing the three-dimensional data to a single dimension, to optimize matching performance.
Datasets
The authors evaluated their approach on three distinct datasets:
- Picking Dataset: A custom dataset focusing on warehouse picking activities, involving grabbing items from shelves. This dataset was collected using smartglasses and a custom wristband.
- Activities of Daily Living (ADL) Dataset: Another custom dataset featuring activities like eating, drinking water, and taking medicine, performed while lying down. This dataset was collected using smartwatches and smartphones.
- CMU-MMAC Dataset: A publicly available, more complex dataset containing kitchen activities, used as a reference.
Key Insights and Ideas
- Leveraging Inertial Data for Video Annotation: The paper demonstrates that accelerometer data from a simple wrist-worn device can be surprisingly effective in identifying and suggesting labels for video recordings, even when the activities are complex.
- Dynamic Time Warping's Suitability: DTW proves to be a robust technique for matching activity templates, effectively handling variations in the speed and duration of human movements across different individuals and recordings.
- The Importance of Granularity: The effectiveness of the approach is highly dependent on the granularity of the activity labels. Simple, distinct motions are easier to template and match than complex activities that involve multiple sub-movements.
- Clustering for Robust Templates: For more complex activities, clustering extracted templates based on their DTW distances can reveal underlying motion similarities, leading to more generalized and robust templates.
- Hand Dominance Matters: In multimodal datasets, differences in performance between left and right hand data can arise, potentially due to natural hand dominance, impacting the consistency of recognition.
Key Results
- Picking Dataset: The method achieved a high overlap (average 69%) for detecting grabbing motions, though accuracy in precise start/end boundaries was a challenge.
- ADL Dataset: The best performing configuration involved using linear acceleration, reducing dimensions to one, and employing a delta-based candidate selection method, achieving a median error of 0.8s and 0.5s for subjects.
- Recall and ROC Curves: For the ADL dataset, recall values up to 0.8 were achieved with appropriate k-values, and ROC curves indicated acceptable performance, particularly for the dominant (right) hand.
- Template Efficiency: The study suggests that a relatively small number of templates (e.g., two) can be sufficient to achieve reasonable results for certain activities.
Limitations and Future Work
- Boundary Accuracy: The paper acknowledges that precisely identifying the start and end boundaries of activities remains a challenge, partly due to the fixed length of templates.
- Activity Granularity: The effectiveness of the approach is sensitive to the granularity of the activity labels. Complex activities might require breaking them down into simpler sub-activities.
- Sensor Modalities: While focusing on acceleration data, the authors suggest that incorporating other sensor modalities could further improve recognition accuracy.
- User Study: A crucial next step is a user study to quantitatively measure the impact of the proposed labeling support on annotation time and agreement among annotators.
- Dataset Enhancement: Expanding the dataset with more instances and a wider range of activities is planned to improve generalization.
Summary updated