Hidden Markov Model-Based Smart Annotation for Benchmark Cyclic Activity Recognition Database Using Wearables

Christine F. Martindale, Sebastijan Sprager, Björn M. Eskofier

TL;DR

This paper introduces a novel "smart annotation" pipeline that significantly reduces the manual effort required to label wearable sensor data for cyclic activities, achieving an annotation effort of just 14%. This pipeline enabled the creation of a large, realistic, and publicly available benchmark dataset of over 150,000 labeled cyclic activities from 80 subjects, crucial for advancing activity recognition research.

What Problem Does This Paper Address?

Accurately recognizing human activities using wearable sensors is a growing field, particularly for applications in healthcare and rehabilitation. However, a major bottleneck is the creation of high-quality, labeled datasets. Specifically, analyzing cyclic activities like walking, running, or jumping at the cycle level (e.g., identifying individual steps or gait phases) is challenging. Existing datasets often suffer from being collected in artificial laboratory settings, lacking sufficient subjects, or failing to capture the nuances of real-world transitions, varying bout durations, and non-straight movements. The sheer manual effort required to meticulously label such detailed data (like individual gait phases) has historically limited the creation of comprehensive benchmarks.

Methods

The core contribution is a "smart annotation pipeline" designed to automate and semi-automate the labeling process. It comprises three main components:

  • Edge Detection: For initial labeling, an edge-detection method on pressure sensor data was used, identifying rising and falling edges corresponding to ground contact. This method was effective but sensitive to data quality.
  • Iterative Hierarchical Hidden Markov Model (hHMM): This is the central piece. An hHMM was trained on manually corrected data. This model then predicted labels for new data, which were subsequently manually corrected. This iterative process, where the model is retrained with newly corrected data, progressively improves its accuracy and reduces the manual correction effort. The hHMM hierarchy was designed to capture both activity and cycle phase information.
  • Local Cyclicity Estimation (LCE) and Peak Detection (PD): For activities where pressure data was unreliable (like cycling), these methods were employed for cycle segmentation. LCE relies on signal cyclicity, while PD uses peak detection in specific sensor axes.

The pipeline also includes post-processing rules to ensure consistency with defined activity and phase labels.

Datasets

The paper introduces a new, publicly available benchmark dataset.

  • FAU-Gait Dataset: This dataset is an extension of a previous, smaller dataset. It comprises approximately 30 hours of data from 80 subjects, recorded in both indoor and outdoor settings. It includes 12 activities, 10 of which are cyclic, and captures transitions, varying bout durations, and non-straight walking. Data was collected using 5 synchronized Inertial Measurement Units (IMUs) on wrists, shoes, and a pocket, along with pressure insoles and video. The dataset is available at www.activitynet.org (video data excluded for privacy).

Key Insights and Ideas

  • Smart Annotation is Key: The paper strongly advocates for and demonstrates the effectiveness of "smart annotation" as a viable solution to the dataset labeling bottleneck, significantly reducing the human effort required.
  • Iterative Learning for Improvement: The iterative training of the hHMM is a powerful concept. By feeding corrected predictions back into the training loop, the model learns from its mistakes and becomes progressively more accurate, leading to a substantial reduction in manual correction over time.
  • Sensor Fusion for Robustness: While the edge detection relied on pressure data, the hHMM leveraged IMU data (accelerometer and gyroscope) from shoe sensors. This highlights the importance of using multiple sensor modalities and choosing the most appropriate ones for different tasks and data quality issues.
  • Realistic Data is Crucial: The emphasis on collecting data in "pseudo-realistic" settings, including transitions and varied bout lengths, is vital for developing algorithms that perform well in real-world applications, not just in controlled lab environments.

Key Results

  • Reduced Annotation Effort: The smart annotation pipeline reduced the manual labeling effort to an average of 14.1% for the entire dataset. For datasets dominated by walking, this effort could be as low as 8%.
  • High F1-Score: The overall F1-score for the pipeline was 89.5%, with an expected F1-score of 93.0% for labeling new data using the trained model.
  • Dataset Size and Diversity: The dataset contains over 150,000 labeled cycles from 80 subjects, covering 12 activities, including transitions and non-straight movements.
  • hHMM Performance: The iterative hHMM method significantly outperformed the edge-detection method for walking, jogging, and running activities, especially for longer steady-state sections, achieving F1-scores over 95% with minimal effort.
  • Accuracy for Specific Activities: For activities like walking, jogging, and running, the hHMM achieved F1-scores of 95.8%, 97.5%, and 92.4% respectively, with very low labeling effort.

Limitations and Future Work

  • Subject Age Range: The dataset primarily includes participants up to 43 years old, which might limit generalizability to older populations.
  • Model Generalizability: While the hHMM is powerful, it still requires initial training data and might benefit from more data or optimized features to further reduce effort for activities like hopping and stairs.
  • Personalized Models: The authors suggest that incorporating personalized models could further lower labeling effort.
  • Real-time and Semi-supervised Approaches: For future applications, they envision real-time or more advanced semi-supervised approaches for task labeling.
  • Broader Activity Scope: While the dataset covers a good range of cyclic activities, future work could expand to include a wider variety of activities and interpretations.

Summary updated

Abstract

Activity monitoring using wearables is becoming ubiquitous, although accurate cycle level analysis, such as step-counting and gait analysis, are limited by a lack of realistic and labeled datasets. The effort required to obtain and annotate such datasets is massive, therefore we propose a smart annotation pipeline which reduces the number of events needing manual adjustment to 14%. For scenarios dominated by walking, this annotation effort is as low as 8%. The pipeline consists of three smart annotation approaches, namely edge detection of the pressure data, local cyclicity estimation, and iteratively trained hierarchical hidden Markov models. Using this pipeline, we have collected and labeled a dataset with over 150,000 labeled cycles, each with 2 phases, from 80 subjects, which we have made publicly available. The dataset consists of 12 different task-driven activities, 10 of which are cyclic. These activities include not only straight and steady-state motions, but also transitions, different ranges of bouts, and changing directions. Each participant wore 5 synchronized inertial measurement units (IMUs) on the wrists, shoes, and in a pocket, as well as pressure insoles and video. We believe that this dataset and smart annotation pipeline are a good basis for creating a benchmark dataset for validation of other semi- and unsupervised algorithms.
BibTeX
@inproceedings{martindale2019hidden,
  author = {Christine F. Martindale and Sebastijan Sprager and Björn M. Eskofier},
  title = {{Hidden Markov Model-Based Smart Annotation for Benchmark Cyclic Activity Recognition Database Using Wearables}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=-djeMnKkTuk},
}

Loading related content...

Figure