A Multi-Task Learning Approach for Human Activity Segmentation and Ergonomics Risk Assessment

Behnoosh Parsa, Ashis G. BanerjeeWACV 2021

TL;DR

This paper introduces a novel multi-task learning framework that simultaneously segments human activities and assesses ergonomics risk from video. By leveraging Graph Convolutional Networks (GCNs) for pose understanding and an Encoder-Decoder Temporal Convolutional Network (ED-TCN) for segmentation, the approach achieves improved ergonomics risk assessment, particularly in industrial settings.

What Problem Does This Paper Address?

Automated Human Activity Evaluation (HAE) is crucial for applications ranging from sports analysis to industrial safety. A significant challenge within HAE is Human Postural Assessment (HPA), which focuses on evaluating the ergonomics risk associated with human posture. Existing HAE methods often struggle with accuracy because they either rely solely on detected skeletons or scene information, failing to capture the intricate correlations between joint movements and body dynamics. Furthermore, many methods are scene-dependent, limiting their generalizability. HPA, in particular, faces challenges due to the task- and object-dependent nature of risk, the cumulative effects of repetitive movements, and individual variations in performing tasks. This paper aims to address these limitations by proposing a unified framework that leverages activity segmentation as an auxiliary task to enhance ergonomics risk assessment.

Methods

The core of this work is a multi-task learning (MTL) framework that combines Human Activity Segmentation (HAS) and Human Postural Assessment (HPA). The framework utilizes a Graph Convolutional Network (GCN) as a backbone to learn rich spatial features from human pose skeletons, capturing inter-joint relationships.

For the Human Activity Segmentation (HAS) task, an Encoder-Decoder Temporal Convolutional Network (ED-TCN) is employed. This architecture is designed to semantically segment long videos into distinct activity classes by modeling long-term dependencies using dilated causal convolutions. This is a novel application of an end-to-end GCN-ED-TCN architecture for HAS.

For the Human Postural Assessment (HPA) task, the GCN features are fed into a Long-Short-Term-Memory (LSTM)-based regressor to predict ergonomics risk scores. The paper explores two architectures: one where the HAS and HPA heads only share GCN features (MTL-base), and another where the softmax output of the HAS head is fused with the GCN features before being fed to the LSTM (MTL-emb). This fusion aims to leverage the activity information to improve risk assessment.

Datasets

The paper evaluates its proposed method on two publicly available datasets:

  • UW-IOM Dataset: This dataset contains 20 videos of industry-relevant activities, with 17 action classes and a four-tier hierarchy for labels. The authors use 3D poses from their earlier work.
  • TUM Kitchen Dataset: This dataset comprises 19 videos of daily kitchen activities. The authors use 3D poses extracted using a method from a previous study.

Key Insights and Ideas

  • Activity Segmentation as an Inductive Bias: The paper demonstrates that using Human Activity Segmentation (HAS) as an auxiliary task significantly improves the performance of Human Postural Assessment (HPA). This suggests that understanding what activity is being performed provides crucial context for assessing the ergonomics risk of a posture.
  • GCNs for Pose-Based Feature Extraction: The study highlights the effectiveness of GCNs in extracting robust spatial features directly from human pose skeletons. This approach is shown to be comparable to, and in some cases better than, methods relying on image-based features for activity segmentation.
  • End-to-End Learning for HAS: The paper presents a novel end-to-end architecture combining GCNs with ED-TCN for Human Activity Segmentation, which learns both spatial and temporal features simultaneously, outperforming previous approaches.
  • Fusion Improves Risk Assessment: The MTL-emb model, which fuses HAS output with GCN features for HPA, shows superior performance in predicting ergonomics risk scores compared to the MTL-base model that only shares GCN features. This underscores the importance of integrating activity context into risk assessment.

Key Results

  • The proposed MTL-emb framework achieves significant improvements in Ergonomics Risk Assessment (ERA) compared to single-task learning approaches on the UW-IOM dataset.
  • For Human Activity Segmentation (HAS) on the UW-IOM dataset, the GCN-ED-TCN approach demonstrates competitive performance against state-of-the-art methods that use pre-trained image-based features.
  • While the method performs well on the UW-IOM dataset, its performance on the TUM Kitchen Dataset is less impressive, particularly for HAS. This is attributed to the similarity of postures in different action classes and potential issues with pose estimation quality in certain views.
  • The paper reports specific metrics like MSE and Spearman's Coefficient for risk assessment and mAP, Edit score, and F1-overlap for activity segmentation, showing quantitative improvements for the MTL approach. For instance, on UW-IOM, MTL-emb achieved a higher Spearman's Coefficient (74.45%) compared to MTL-base (66.68%).

Limitations and Future Work

  • Generalizability: The authors acknowledge that generalization to unseen activities remains a challenge, as the model is trained on labeled data.
  • Pose Information: The current model primarily uses joint positions. Future work could incorporate other kinematic information like velocity and acceleration, which are known to be important for injury prediction.
  • TUM Dataset Performance: The model's weaker performance on the TUM dataset suggests that for highly similar poses or poor pose estimation, incorporating image-based features might be beneficial.
  • Future Directions: The authors suggest developing biomechanics-based human pose representation models that learn causal relationships between joint kinematics and ergonomics risk. They also believe the proposed multi-task framework can be extended to other action and skill assessment problems.

Summary updated

BibTeX
@inproceedings{parsa2021multi,
  author = {Behnoosh Parsa and Ashis G. Banerjee},
  title = {{A Multi-Task Learning Approach for Human Activity Segmentation and Ergonomics Risk Assessment}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure