Active Generation Network of Human Skeleton for Action Recognition

Long Liu, Xin Wang, Fangming Li, Jiayu ChenarXiv 2024 · OpenReview

TL;DR

This paper introduces the Active Generation Network (AGN), a novel approach for generating realistic human skeleton action data, especially when training data is scarce. AGN uses motion style transfer to create new actions while an uncertainty metric network intelligently selects the most informative generated samples to improve overall data quality and action recognition performance.

What Problem Does This Paper Address?

Human Action Recognition (HAR) is a crucial area in computer vision, with skeleton-based methods being a popular choice due to their robustness. However, a significant challenge in training effective skeleton-based HAR models is the scarcity of data. Obtaining large, diverse datasets is difficult due to privacy concerns, low occurrence rates of certain actions, and high collection costs. Existing data generation techniques often struggle with temporal consistency, lack diversity, and perform poorly when only a few samples are available. This paper aims to address this data sparsity problem by proposing a method that can generate high-quality, diverse human action data from limited examples.

Methods

The proposed Active Generation Network (AGN) is composed of two main components:

  • Motion Generation Network (MGN): This network is responsible for generating new human action sequences. It utilizes a spatio-temporal graph convolutional network (ST-GCN) as its backbone. The MGN employs a motion style transfer approach, where it learns to implicitly capture the morphological features of a target action while preserving the category features of a source action. This is achieved through an encoder-decoder architecture that incorporates graph upsampling and downsampling to extract and fuse multi-scale spatio-temporal features. A key aspect is the use of Body Part Adaptive Instance Normalisation (BP-AdaIN) and Body Part Attention Network (BP-ATN) in the decoder to enhance feature fusion and local/global feature extraction.
  • Uncertainty Metric Network (UMN): This component acts as a guide for the MGN, ensuring the quality of the generated actions. It leverages active learning principles. First, an action recognition model (ST-GCN) is trained on the limited available data to generate prediction vectors for newly generated actions. An uncertainty metric is then applied to these predictions to score their reliability. Actions with higher uncertainty are considered more informative and are selected to augment the training set, guiding the MGN to produce better results in subsequent iterations. This iterative process helps in selecting the most valuable samples from the generated data.

The training of the MGN involves a multi-objective loss function, including reconstruction loss, cycle consistency loss, and a feature triplet loss to encourage clustering of similar actions and separation of different ones in the latent space.

Datasets

The primary dataset used in this paper is NTU-RGB+D 60. The authors define three subsets:

  • Mfull: The complete dataset.
  • Mseen: Contains six categories of "seen" actions ("Brush Teeth", "Pick up", "Reading", "Take off a Hat", "Kicking Something", and "Sneeze").
  • Munseen: Contains six categories of "unseen" actions ("Drink Water", "Throw", "Sit Down", "Clapping", "Jump up", and "Bow").

Key Insights and Ideas

  • Active Learning for Data Generation: The core innovation is the integration of active learning principles into the data generation process. Instead of blindly generating data, the UMN actively selects the most informative generated samples, significantly improving the quality and utility of the synthetic data.
  • Motion Style Transfer for Skeleton Data: The paper effectively adapts motion style transfer techniques, typically used for images, to the domain of human skeleton data. This allows for the generation of novel actions by combining the characteristics of existing ones.
  • Handling Data Scarcity: The AGN is specifically designed to address the challenge of limited training data, demonstrating that high-quality action data can be generated even from a single or a few samples.
  • Preserving Source and Target Features: The MGN's architecture is designed to balance preserving the semantic meaning (category features) of the source action with the physical motion characteristics (morphological features) of the target action, leading to more realistic and meaningful generated actions.

Key Results

  • Qualitative Evaluation: Visualizations show that the generated actions are temporally consistent and retain both source and target action features, even for unseen actions. The generated actions are described as more natural and coordinated compared to previous methods like Motion Puzzle.
  • Quantitative Evaluation (FMD & Accuracy):
    • On seen actions, the generated data achieved an average accuracy of 94.42% and an average FMD of 2.11.
  • On unseen actions, the average accuracy was 92.18% and the average FMD was 2.67.
  • Action Recognition Performance: When using generated data to augment a small training set (1% of Mtrain), the AGN achieved a top-1 accuracy of 92.18%, demonstrating that the generated data is highly effective for improving action recognition performance, with only a small drop compared to using a larger portion of real data.
  • Ablation Study: The inclusion of the triplet loss (Ltrip) improved the generation quality, and the UMN component significantly boosted the recognition accuracy by 4.76% compared to the MGN without UMN.

Limitations and Future Work

  • Limitations: The paper acknowledges that when sampling only 1% of the data for generation, the generative model has limited source data distribution learning, leading to a larger deviation from the original complete data distribution. While the generated data is good, it's not a perfect replacement for large amounts of real data in all scenarios.
  • Future Work: The paper implicitly suggests that further improvements could be made by exploring more sophisticated uncertainty metrics or by developing more advanced generative models that can better capture the nuances of human motion. The effectiveness of the UMN in guiding generation also opens avenues for exploring its application in other generative tasks where data scarcity is an issue.

Summary updated

Abstract

Data generation is a data augmentation technique for enhancing the generalization ability for skeleton-based human action recognition. Most existing data generation methods face challenges to ensure the temporal consistency of the dynamic information for action. In addition, the data generated by these methods lack diversity when only a few training samples are available. To solve those problems, We propose a novel active generative network (AGN), which can adaptively learn various action categories by motion style transfer to generate new actions when the data for a particular action is only a single sample or few samples. The AGN consists of an action generation network and an uncertainty metric network. The former, with ST-GCN as the Backbone, can implicitly learn the morphological features of the target action while preserving the category features of the source action. The latter guides generating actions. Specifically, an action recognition model generates prediction vectors for each action, which is then scored using an uncertainty metric. Finally, UMN provides the uncertainty sampling basis for the generated actions.
BibTeX
@misc{liu2024active,
  author = {Long Liu and Xin Wang and Fangming Li and Jiayu Chen},
  title = {{Active Generation Network of Human Skeleton for Action Recognition}},
  year = {2024},
  eprint = {2401.17086},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.17086},
}
Figure