Smart Annotation of Cyclic Data Using Hierarchical Hidden Markov Models

Christine F. Martindale, Florian Hoenig, Christina Strohrmann, Björn M. Eskofier

TL;DR

This paper introduces a "smart annotation" method using hierarchical hidden Markov models (hHMMs) to significantly reduce the cost of labeling cyclic sensor data, like human motion. By leveraging semi-supervised learning and a known cycle count, the method achieves comparable accuracy to fully-supervised approaches with substantially less manual effort, paving the way for more accessible analysis of everyday activities.

What Problem Does This Paper Address?

Analyzing cyclic sensor data, such as human movement (walking, running) or biological signals (heart activity), is crucial for applications in healthcare, sports, and rehabilitation. However, accurately segmenting this data into individual cycles is a prerequisite for detailed analysis. Traditional methods for this segmentation often rely on fully-supervised learning, which demands extensive manual annotation of each cycle in the training data. This manual labeling is time-consuming, expensive, and prone to human error. While semi-supervised and unsupervised methods exist, they often require more complex setups (multiple sensors, body models) or don't address the specific challenge of segmenting individual cycles within activities. This paper aims to bridge this gap by developing a method that drastically reduces annotation cost while maintaining high accuracy for cyclic data segmentation.

Methods

The core of the proposed method is a smart annotation (SA) pipeline built upon hierarchical hidden Markov models (hHMMs). The process begins with an initial, "naive" linear segmentation of data segments that are known to contain a specific number of cycles of a single activity class. This initial segmentation is then used for semi-supervised training of an hHMM using Viterbi training. This training is iterative:

  1. Linear Alignment: Data is divided into segments based on a known cycle count.
  2. Semi-supervised HMM Training: An hHMM is trained on these linearly segmented data.
  3. Forced Alignment: The trained hHMM is used to find the most probable sequence of states that matches the known number of cycles. This step refines the segmentation boundaries.
  4. Supervised Training: The refined segmentation boundaries from the forced alignment are then used to train the HMM in a fully-supervised manner for subsequent iterations.

This iterative process, where the output of one step informs the next, allows the model to learn accurate segmentation boundaries with minimal initial supervision (only the cycle count and class are needed). The paper also extends this SA model for simultaneous smart annotation and classification of continuous, unseen data, where cycle numbers are not known beforehand.

Datasets

The study utilized a dataset of cyclic activities, specifically walking and running, collected from 18 participants using a single inertial measurement unit (IMU) worn on the left ankle. The data was collected "in the wild" in an outdoor setting and was simultaneously video recorded for annotation purposes. Reference labels were generated by human annotators through the synchronized analysis of video and sensor data. The dataset included segments of walking, running, and standing (labeled as rest). No new benchmark datasets were introduced; instead, existing data was used to demonstrate the method's applicability.

Key Insights and Ideas

  • Annotation Cost Reduction is Key: The primary insight is that by leveraging a known cycle count, even for short data segments, the annotation burden can be reduced by an order of magnitude (approximately 10x) compared to full manual segmentation.
  • hHMMs for Hierarchical Structure: The use of hHMMs is crucial, allowing the model to capture the hierarchical nature of cyclic data (e.g., states within a cycle, transitions between cycles) more effectively than standard HMMs.
  • Semi-Supervised Learning as a Bridge: The method effectively bridges the gap between fully-supervised and unsupervised learning, offering a practical solution that requires significantly less manual effort than traditional supervised approaches.
  • Robustness to Noise and Inconsistency: The results suggest the method is robust to noisy annotations and variations in labeling, as human annotator inconsistencies did not drastically impact the overall performance.
  • Generalizability Beyond Gait: The method's reliance on the cyclic nature of data rather than specific body models or sensor placements suggests its potential applicability to a wide range of cyclic sensor data and modalities.

Key Results

  • Smart Annotation Pipeline: The best smart annotation model (SA-GZ4) achieved a mean absolute error in segmentation of stride borders (SB) of 0.041 ± 0.020 s and a mean absolute cycle duration error of 0.032 ± 0.005 s. This represents an average error of about 4.0% of an individual cycle duration.
  • Simultaneous Segmentation and Classification: When applied to simultaneous segmentation and classification, the SA-GZ4 model achieved an overall F1-score of 93.2%. This is comparable to a fully-supervised model (Sup-AXGZ8) which achieved 96.8%, but with significantly lower annotation cost.
  • Comparison to Supervised Methods: The smart annotation approach yielded results comparable to fully-supervised methods, with F1-scores differing by less than 1.0% in some cases, demonstrating its effectiveness.
  • Low Misclassification: Misclassification between walking and running was observed, but generally remained low (e.g., less than 1% in some confusion matrices).

Limitations and Future Work

  • Cycle Count Requirement: A current limitation is the need for the number of cycles per data sample and the class for training. While this is a lower annotation cost than full segmentation, it's still a form of supervision.
  • Automatic Cycle Detection: The authors propose investigating automatic detection of the number of cycles and class for input data as a next step, moving closer to zero-resource learning.
  • Clustering for Class Problem: A clustering method could be explored to solve the class identification problem for input data.
  • Noisy Data and Mixed Cases: The method's robustness to noisy labels suggests potential for application in scenarios with noisy data or mixed labeled and unlabeled data.
  • Internal State Interpretation: Further investigation into the meaning of internal states within the hHMM could lead to more detailed cycle and phase segmentation.
  • Dataset Size and Sensor Modalities: Future work includes testing the method on larger datasets and exploring its application with different sensor modalities and positions.

Summary updated

BibTeX
@inproceedings{martindale2017smart,
  author = {Christine F. Martindale and Florian Hoenig and Christina Strohrmann and Björn M. Eskofier},
  title = {{Smart Annotation of Cyclic Data Using Hierarchical Hidden Markov Models}},
  booktitle = {Sensors},
  year = {2017},
  url = {https://openreview.net/forum?id=EBiincZl12S},
}
Figure