Abstract
There are two competing standards for self-supervised learning in action recognition from 3D skeletons. Su et al., 2020 [31] used an auto-encoder architecture and an image reconstruction objective function to achieve state-of-the-art performance on the NTU60 C-View benchmark. Rao et al., 2020 [23] used Contrastive learning in the latent space to achieve state-of-the-art performance on the NTU60 C-Sub benchmark. Here, we reconcile these disparate approaches by developing a taxonomy of self-supervised learning for action recognition. We observe that leading approaches generally use one of two types of objective functions: those that seek to reconstruct the input from a latent representation ("Attractive" learning) versus those that also try to maximize the representations distinctiveness ("Contrastive" learning). Independently, leading approaches also differ in how they implement these objective functions: there are those that optimize representations in the decoder output space and those which optimize representations in the network’s latent space (encoder output). We find that combining these approaches leads to larger gains in performance and tolerance to transformation than is achievable by any individual method, leading to state-of-the-art performance on three standard action recognition datasets. We include links to our code and data.
Introduction
Training deep learning models for 3D skeleton-based action recognition typically requires massive amounts of labeled video data, which is both expensive and time-consuming to acquire. Self-supervised learning (SSL) offers a promising alternative by allowing models to learn from unlabeled data through pretext tasks. However, the field has been fragmented between two primary paradigms: Auto-encoders that focus on input reconstruction and Contrastive learning methods that focus on distinguishing between samples.
This work, presented at WACV 2022, provides a systematic taxonomy to unify these disparate methods. By analyzing what is learned and where it is learned, the authors demonstrate that the optimal strategy involves a hybrid approach that integrates both reconstruction and contrastive objectives across the entire network architecture.
A Unified Taxonomy for SSL
The proposed framework categorizes SSL methods along two orthogonal dimensions:
1. The Objective Function ("What")
- Attractive SSL: The model learns to pull representations of similar data (e.g., a skeleton sequence and its augmented version) together. This is the hallmark of standard reconstruction tasks.
- Contrastive SSL: The model not only pulls similar representations together but also actively pushes dissimilar representations (different action samples) apart in the feature space.
2. The Supervision Location ("Where")
- Encoder Supervision: The learning objective is applied to the latent "bottleneck" space (the output of the encoder).
- Decoder Supervision: The learning objective is applied to the output of the decoder, typically involving the reconstruction of the original skeleton sequence.
Methodology and Results
The researchers systematically evaluated various combinations of these dimensions using a simple 1-layer Gated Recurrent Unit (GRU) architecture. Their optimal configuration, denoted as Ours, combines an Attractive Decoder (standard reconstruction) with an Attractive + Contrastive Encoder.
Key Findings
- Synergistic Effects: Encoder and Decoder objectives are highly synergistic. Models optimizing both consistently outperform those restricted to a single location.
- Decoder Dominance: On average, models utilizing Decoder-based reconstruction objectives achieved higher accuracy than those relying solely on latent space (Encoder) learning.
- Transformation Tolerance: The hybrid model inherits specific strengths from its components: Contrastive Encoder learning provides high tolerance to rotations, while the Decoder objective is essential for handling shearing transformations.
- Architectural Simplicity: Despite using a basic GRU backbone, the method outperformed more complex Graph Convolutional Networks (GCNs) that utilized less effective SSL strategies, highlighting that the learning objective is often more critical than the backbone architecture.
Performance Benchmarks
The hybrid approach established new state-of-the-art (SOTA) results for SSL methods across three major datasets:
- NTU-60: Achieved 76.3% (Cross-View) and 67.0% (Cross-Subject).
- NTU-120: Delivered an improvement of over 10% compared to previous benchmarks.
- NW-UCLA: Reached 86.08% accuracy, significantly outperforming prior SSL methods.
Conclusion
By taxonomizing SSL for 3D action recognition, this research identifies a clear path forward for the field. The combination of reconstruction in the decoder and contrastive learning in the encoder provides a robust, high-performance framework that is tolerant to various data transformations. Future work will explore scaling these objectives to more advanced architectures like Transformers and further closing the gap with fully supervised models.
Summary updated