Self-Supervised Action Representation Learning Based on Asymmetric Skeleton Data Augmentation

Hualing Zhou, Xi Li, Dahong Xu, Hong Liu, Jianping Guo, Yihan ZhangSensors 2022 · DOI

TL;DR

This paper introduces an "asymmetric data augmentation" strategy for self-supervised learning of human action representations from skeleton data. By applying different augmentation techniques to distinct "views" of the skeleton (joint, motion, bone) and using a novel pipeline, they significantly improve the quality and robustness of learned features, leading to state-of-the-art performance on benchmark datasets.

What Problem Does This Paper Address?

Learning to recognize human actions from skeleton data is a crucial task in computer vision, with applications ranging from human-computer interaction to sports analysis. While supervised methods have achieved good results, they require extensive, manually labeled data, which is costly and time-consuming to acquire. Self-supervised learning offers a promising alternative by leveraging unlabeled data. A key component of self-supervised methods, particularly contrastive learning, is data augmentation, which generates positive sample pairs. However, existing methods often use simple augmentations that don't fully capture the rich information in skeleton data, leading to insufficient feature learning and impacting downstream task performance. This paper aims to address this by designing a more sophisticated data augmentation strategy tailored for skeleton data.

Methods

The core contribution of this paper is the asymmetric data augmentation pipeline. This pipeline is designed to create more challenging and informative positive sample pairs for contrastive learning.

Here's a breakdown of the approach:

  • Asymmetric Pipeline: The pipeline consists of two branches (left and right). Each branch applies a different set of augmentation techniques. This asymmetry is key to increasing the "convergence difficulty" of the contrastive learning task, forcing the model to learn more robust features.
  • View-Specific Augmentation: The authors recognize that different "views" of skeleton data (joint positions, motion/temporal changes, and bone lengths) have distinct characteristics. They strategically choose augmentation methods that are best suited for each view. For instance, "crop" is used for motion augmentation, while "shear" is applied to bone augmentation.
  • Random Probability Activation: To avoid overly distorting the original skeleton data, especially for the joint view, they employ random probability activation for certain augmentation methods. This ensures that augmented samples remain sufficiently similar to the original while still introducing meaningful variations.
  • Contrastive Learning Framework: The proposed augmentation strategy is integrated into a self-supervised contrastive learning framework, specifically inspired by MoCo v2. This framework uses an encoder-decoder structure and a momentum encoder to learn representations by maximizing agreement between augmented views of the same sample and minimizing agreement with other samples.
  • Multi-View Extension: The authors also extend their approach to a composite framework that utilizes multiple skeleton views (joint, motion, and bone) simultaneously. This composite framework leverages cross-view consistency and a "high-confidence knowledge mining" mechanism to further enhance representation learning.

Datasets

The paper evaluates its proposed method on two widely used large-scale 3D skeleton action recognition datasets:

  • NTU RGB + D 60 (NTU-60): Contains 60 action classes and 56,880 skeleton sequences.
  • NTU RGB + D 120 (NTU-120): An extension of NTU-60, featuring 120 action classes and 113,945 skeleton sequences.

Both datasets are evaluated using standard cross-subject (xsub) and cross-view (xview) protocols.

Key Insights and Ideas

  • Augmentation is Not One-Size-Fits-All: The paper strongly emphasizes that simple, generic data augmentation strategies are insufficient for complex data like 3D skeletons. Tailoring augmentations to the specific characteristics of different data representations (views) is crucial.
  • Asymmetry Drives Learning: The core idea of an "asymmetric" augmentation pipeline is to create a more challenging learning objective. By having different transformations in each branch, the model is forced to learn more invariant and robust features.
  • Balancing Augmentation and Similarity: The authors highlight the trade-off between creating diverse augmented samples and maintaining enough similarity to the original for effective contrastive learning. Their random probability activation strategy is a clever way to navigate this balance.
  • Multi-View Synergy: Combining information from different skeleton views (joint, motion, bone) and using cross-view consistency can significantly boost performance, especially when coupled with effective augmentation.

Key Results

The proposed asymmetric data augmentation strategy demonstrates significant improvements across various experiments:

  • Ablation Studies (NTU-60):
    • The proposed method achieved 79.0% xsub and 84.6% xview accuracy.
  • This represents an improvement of 1.2% over basic augmentation and outperforms "extreme augmentation" (using all seven methods simultaneously).
  • Removing specific augmentation methods in the left branch showed that each contributed to the overall performance, with random activation of Gaussian blur, noise, joint mask, and channel mask being particularly effective.
  • Linear Evaluation (NTU-60):
    • The method achieved 79.0% xsub and 84.6% xview accuracy, outperforming state-of-the-art methods like 3s-SkeletonCLR and 3s-CrosSCLR by a notable margin (e.g., 4.0% and 4.8% on xsub and xview respectively compared to 3s-SkeletonCLR).
  • Linear Evaluation (NTU-120):
    • The method achieved 68.5% xsub and 69.9% xset accuracy, also outperforming existing methods, including 3s-CrosSCLR by 0.6% and 3.2% on xsub and xset respectively.

The paper also shows that their method leads to lower contrastive loss curves during training, indicating more effective learning.

Limitations and Future Work

The authors acknowledge a few limitations and suggest future research directions:

  • Limited Views: The current approach augments only three specific skeleton views (joint, motion, bone). They suggest that exploring and augmenting more views could potentially lead to even better results.
  • Framework and Network: The current work uses a basic contrastive learning framework and feature extraction network. Future work could explore integrating their augmentation strategy with more advanced contrastive learning frameworks and feature extraction architectures to further push performance boundaries.
  • Robustness: Investigating how to make learned feature representations even more robust through richer skeleton information and advanced augmentation techniques is a promising avenue.

Summary updated

Abstract

Contrastive learning has received increasing attention in the field of skeleton-based action representations in recent years. Most contrastive learning methods use simple augmentation strategies to construct pairs of positive samples. When using such pairs of positive samples to learn action representations, deeper feature information cannot be learned, thus affecting the performance of downstream tasks. To solve the problem of insufficient learning ability, we propose an asymmetric data augmentation strategy and attempt to apply it to the training of 3D skeleton-based action representations. First, we carefully study the different characteristics presented by different skeleton views and choose a specific augmentation method for a certain view. Second, specific augmentation methods are incorporated into the left and right branches of the asymmetric data augmentation pipeline to increase the convergence difficulty of the contrastive learning task, thereby significantly improving the quality of the learned action representations. Finally, since many methods directly act on the joint view, the augmented samples are quite different from the original samples. We use random probability activation to transform the joint view to avoid extreme augmentation of the joint view. Extensive experiments on NTU RGB + D datasets show that our method is effective.
BibTeX
@inproceedings{zhou2022self,
  author = {Hualing Zhou and Xi Li and Dahong Xu and Hong Liu and Jianping Guo and Yihan Zhang},
  title = {{Self-Supervised Action Representation Learning Based on Asymmetric Skeleton Data Augmentation}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s22228989},
}
Figure