Abstract
Training deep learning models for 3D skeleton-based action recognition typically requires vast amounts of manually labeled data. To address this, we present a novel self-supervised learning (SSL) framework named MCC (Motion Consistency and Continuity). By leveraging two pretext tasks—recognizing the same action at varying speeds (Motion Consistency) and predicting missing frames in a sequence (Motion Continuity)—the framework enables models to learn high-quality movement representations from unlabeled data. Experimental results across major benchmarks like NTU RGB+D and Kinetics-Skeleton demonstrate that MCC significantly narrows the gap between unsupervised and supervised performance, offering superior efficiency in semi-supervised and transfer learning scenarios.
Introduction
3D skeleton-based action recognition, which represents human movement through a series of 3D joint coordinates, is a robust method for behavior analysis. However, the high cost of manual labeling for large-scale datasets remains a significant bottleneck. While Self-Supervised Learning (SSL) has seen success in image and video domains, applying it to skeletal data is challenging. Skeletons lack rich appearance features like texture or color, and models often struggle with "playback speed ambiguity"—the difficulty in distinguishing between a slow version of one action and a fast version of another. Without a fundamental understanding of motion dynamics, models frequently converge on trivial solutions, memorizing individual frames rather than learning the underlying action.
The MCC Framework
The proposed MCC framework introduces two primary pretext tasks designed to teach models the intrinsic "physics" of human movement without human supervision. The framework is backbone-agnostic, compatible with various architectures such as ST-GCN, 2S-AGCN, and AS-GCN.
1. Spatial-Motion Consistency
To address speed ambiguity, the model is trained using a triplet-based approach. It is presented with three versions of a clip:
- The Anchor (Basic): A standard action sequence.
- The Positive (Speed-changed): The same action played at a different speed (e.g., 2x fast-forward).
- The Negative (Motion-broken): The same frames as the anchor, but shuffled randomly in time.
By forcing the model to pull the "Positive" closer in its latent space while pushing the "Negative" away, the model learns that the essence of an action remains constant regardless of speed, but is destroyed if temporal order is lost. Shuffling serves as a strong negative, forcing the model to capture fine-grained temporal structures.
2. Temporal-Motion Continuity
The second task is based on the intuition that a model truly understanding motion should be able to "imagine" intermediate states. Unlike standard auto-encoding which reconstructs the entire input, MCC uses an interpolation module. The model receives a sparse version of a clip (e.g., every other frame) and must reconstruct the missing frames to create a coherent, natural motion sequence.
This interpolation task forces the model to capture the "flow" and continuity between adjacent frames, which is more representative of human perception than simple frame-by-frame reconstruction.
Experimental Evaluation
The researchers evaluated MCC on four major benchmarks: NTU RGB+D 60 & 120, Kinetics-Skeleton, and PKUMMD.
Key Results
- State-of-the-Art Performance: MCC outperformed previous self-supervised methods, including LongT GAN and MS2L, across all datasets.
- Semi-Supervised Efficiency: In scenarios with limited labels (5% or 10%), the MCC pre-trained model provided a massive accuracy boost. For instance, it achieved a +4.3% improvement on NTU-60 with only 5% of labels compared to training from scratch.
- Generalizability: Models pre-trained on the large NTU dataset and fine-tuned on the smaller PKUMMD dataset showed significant gains, proving that the learned motion features transfer effectively across different environments and camera setups.
Conclusion and Future Work
The MCC framework demonstrates that explicitly training for speed invariance and temporal continuity allows models to learn robust skeletal representations without manual labels. While highly effective for single-person actions, future research could extend this approach to complex multi-person interactions (e.g., hugging). Additionally, as the field moves toward Transformer-based architectures, testing the MCC framework on these newer backbones represents a promising next step.
Summary updated