TL;DR
This paper introduces an "asymmetric data augmentation" strategy for self-supervised learning of human action representations from skeleton data. By applying different augmentation techniques to distinct "views" of the skeleton (joint, motion, bone) and using a novel pipeline, they significantly improve the quality and robustness of learned features, leading to state-of-the-art performance on benchmark datasets.
What Problem Does This Paper Address?
Learning to recognize human actions from skeleton data is a crucial task in computer vision, with applications ranging from human-computer interaction to sports analysis. While supervised methods have achieved good results, they require extensive, manually labeled data, which is costly and time-consuming to acquire. Self-supervised learning offers a promising alternative by leveraging unlabeled data. A key component of self-supervised methods, particularly contrastive learning, is data augmentation, which generates positive sample pairs. However, existing methods often use simple augmentations that don't fully capture the rich information in skeleton data, leading to insufficient feature learning and impacting downstream task performance. This paper aims to address this by designing a more sophisticated data augmentation strategy tailored for skeleton data.
Methods
The core contribution of this paper is the asymmetric data augmentation pipeline. This pipeline is designed to create more challenging and informative positive sample pairs for contrastive learning.
Here's a breakdown of the approach:
- Asymmetric Pipeline: The pipeline consists of two branches (left and right). Each branch applies a different set of augmentation techniques. This asymmetry is key to increasing the "convergence difficulty" of the contrastive learning task, forcing the model to learn more robust features.
- View-Specific Augmentation: The authors recognize that different "views" of skeleton data (joint positions, motion/temporal changes, and bone lengths) have distinct characteristics. They strategically choose augmentation methods that are best suited for each view. For instance, "crop" is used for motion augmentation, while "shear" is applied to bone augmentation.
- Random Probability Activation: To avoid overly distorting the original skeleton data, especially for the joint view, they employ random probability activation for certain augmentation methods. This ensures that augmented samples remain sufficiently similar to the original while still introducing meaningful variations.
- Contrastive Learning Framework: The proposed augmentation strategy is integrated into a self-supervised contrastive learning framework, specifically inspired by MoCo v2. This framework uses an encoder-decoder structure and a momentum encoder to learn representations by maximizing agreement between augmented views of the same sample and minimizing agreement with other samples.
- Multi-View Extension: The authors also extend their approach to a composite framework that utilizes multiple skeleton views (joint, motion, and bone) simultaneously. This composite framework leverages cross-view consistency and a "high-confidence knowledge mining" mechanism to further enhance representation learning.
Datasets
The paper evaluates its proposed method on two widely used large-scale 3D skeleton action recognition datasets:
- NTU RGB + D 60 (NTU-60): Contains 60 action classes and 56,880 skeleton sequences.
- NTU RGB + D 120 (NTU-120): An extension of NTU-60, featuring 120 action classes and 113,945 skeleton sequences.
Both datasets are evaluated using standard cross-subject (xsub) and cross-view (xview) protocols.
Key Insights and Ideas
- Augmentation is Not One-Size-Fits-All: The paper strongly emphasizes that simple, generic data augmentation strategies are insufficient for complex data like 3D skeletons. Tailoring augmentations to the specific characteristics of different data representations (views) is crucial.
- Asymmetry Drives Learning: The core idea of an "asymmetric" augmentation pipeline is to create a more challenging learning objective. By having different transformations in each branch, the model is forced to learn more invariant and robust features.
- Balancing Augmentation and Similarity: The authors highlight the trade-off between creating diverse augmented samples and maintaining enough similarity to the original for effective contrastive learning. Their random probability activation strategy is a clever way to navigate this balance.
- Multi-View Synergy: Combining information from different skeleton views (joint, motion, bone) and using cross-view consistency can significantly boost performance, especially when coupled with effective augmentation.
Key Results
The proposed asymmetric data augmentation strategy demonstrates significant improvements across various experiments:
- Ablation Studies (NTU-60):
- The proposed method achieved 79.0% xsub and 84.6% xview accuracy.
- This represents an improvement of 1.2% over basic augmentation and outperforms "extreme augmentation" (using all seven methods simultaneously).
- Removing specific augmentation methods in the left branch showed that each contributed to the overall performance, with random activation of Gaussian blur, noise, joint mask, and channel mask being particularly effective.
- Linear Evaluation (NTU-60):
- The method achieved 79.0% xsub and 84.6% xview accuracy, outperforming state-of-the-art methods like 3s-SkeletonCLR and 3s-CrosSCLR by a notable margin (e.g., 4.0% and 4.8% on xsub and xview respectively compared to 3s-SkeletonCLR).
- Linear Evaluation (NTU-120):
- The method achieved 68.5% xsub and 69.9% xset accuracy, also outperforming existing methods, including 3s-CrosSCLR by 0.6% and 3.2% on xsub and xset respectively.
The paper also shows that their method leads to lower contrastive loss curves during training, indicating more effective learning.
Limitations and Future Work
The authors acknowledge a few limitations and suggest future research directions:
- Limited Views: The current approach augments only three specific skeleton views (joint, motion, bone). They suggest that exploring and augmenting more views could potentially lead to even better results.
- Framework and Network: The current work uses a basic contrastive learning framework and feature extraction network. Future work could explore integrating their augmentation strategy with more advanced contrastive learning frameworks and feature extraction architectures to further push performance boundaries.
- Robustness: Investigating how to make learned feature representations even more robust through richer skeleton information and advanced augmentation techniques is a promising avenue.
Summary updated