Self-supervised 3D Skeleton Action Representation Learning with Motion Consistency and Continuity

Yukun Su, Guosheng Lin, Qingyao Wu

Abstract

Training deep learning models for 3D skeleton-based action recognition typically requires vast amounts of manually labeled data. To address this, we present a novel self-supervised learning (SSL) framework named MCC (Motion Consistency and Continuity). By leveraging two pretext tasks—recognizing the same action at varying speeds (Motion Consistency) and predicting missing frames in a sequence (Motion Continuity)—the framework enables models to learn high-quality movement representations from unlabeled data. Experimental results across major benchmarks like NTU RGB+D and Kinetics-Skeleton demonstrate that MCC significantly narrows the gap between unsupervised and supervised performance, offering superior efficiency in semi-supervised and transfer learning scenarios.

Introduction

3D skeleton-based action recognition, which represents human movement through a series of 3D joint coordinates, is a robust method for behavior analysis. However, the high cost of manual labeling for large-scale datasets remains a significant bottleneck. While Self-Supervised Learning (SSL) has seen success in image and video domains, applying it to skeletal data is challenging. Skeletons lack rich appearance features like texture or color, and models often struggle with "playback speed ambiguity"—the difficulty in distinguishing between a slow version of one action and a fast version of another. Without a fundamental understanding of motion dynamics, models frequently converge on trivial solutions, memorizing individual frames rather than learning the underlying action.

The MCC Framework

The proposed MCC framework introduces two primary pretext tasks designed to teach models the intrinsic "physics" of human movement without human supervision. The framework is backbone-agnostic, compatible with various architectures such as ST-GCN, 2S-AGCN, and AS-GCN.

1. Spatial-Motion Consistency

To address speed ambiguity, the model is trained using a triplet-based approach. It is presented with three versions of a clip:

  • The Anchor (Basic): A standard action sequence.
  • The Positive (Speed-changed): The same action played at a different speed (e.g., 2x fast-forward).
  • The Negative (Motion-broken): The same frames as the anchor, but shuffled randomly in time.
The Spatial-Motion Consistency task uses a triplet loss to pull speed-invariant representations together while pushing shuffled, broken motion away.

By forcing the model to pull the "Positive" closer in its latent space while pushing the "Negative" away, the model learns that the essence of an action remains constant regardless of speed, but is destroyed if temporal order is lost. Shuffling serves as a strong negative, forcing the model to capture fine-grained temporal structures.

2. Temporal-Motion Continuity

The second task is based on the intuition that a model truly understanding motion should be able to "imagine" intermediate states. Unlike standard auto-encoding which reconstructs the entire input, MCC uses an interpolation module. The model receives a sparse version of a clip (e.g., every other frame) and must reconstruct the missing frames to create a coherent, natural motion sequence.

The Temporal-Motion Continuity task involves an encoder-decoder architecture that interpolates missing frames from a sparse input sequence.

This interpolation task forces the model to capture the "flow" and continuity between adjacent frames, which is more representative of human perception than simple frame-by-frame reconstruction.

Experimental Evaluation

The researchers evaluated MCC on four major benchmarks: NTU RGB+D 60 & 120, Kinetics-Skeleton, and PKUMMD.

Key Results

  • State-of-the-Art Performance: MCC outperformed previous self-supervised methods, including LongT GAN and MS2L, across all datasets.
  • Semi-Supervised Efficiency: In scenarios with limited labels (5% or 10%), the MCC pre-trained model provided a massive accuracy boost. For instance, it achieved a +4.3% improvement on NTU-60 with only 5% of labels compared to training from scratch.
  • Generalizability: Models pre-trained on the large NTU dataset and fine-tuned on the smaller PKUMMD dataset showed significant gains, proving that the learned motion features transfer effectively across different environments and camera setups.

Conclusion and Future Work

The MCC framework demonstrates that explicitly training for speed invariance and temporal continuity allows models to learn robust skeletal representations without manual labels. While highly effective for single-person actions, future research could extend this approach to complex multi-person interactions (e.g., hugging). Additionally, as the field moves toward Transformer-based architectures, testing the MCC framework on these newer backbones represents a promising next step.

Summary updated

Abstract

Recently, self-supervised learning (SSL) has been proved very effective and it can help boost the performance in learning representations from unlabeled data in the image domain. Yet, very little is explored about its usefulness in 3D skeleton-based action recognition understanding. Directly applying existing SSL techniques for 3D skeleton learning, however, suffers from trivial solutions and imprecise representations. To tackle these drawbacks, we consider perceiving the consistency and continuity of motion at different playback speeds are two critical issues. To this end, we propose a novel SSL method to learn the 3D skeleton representation in an efficacious way. Specifically, by constructing a positive clip (speed-changed) and a negative clip (motion-broken) of the sampled action sequence, we encourage the positive pairs closer while pushing the negative pairs to force the network to learn the intrinsic dynamic motion consistency information. Moreover, to enhance the learning features, skeleton interpolation is further exploited to model the continuity of human skeleton data. To validate the effectiveness of the proposed method, extensive experiments are conducted on Kinetics, NTU60, NTU120, and PKUMMD datasets with several alternative network architectures. Experimental evaluations demonstrate the superiority of our approach and through which, we can gain significant performance improvement without using extra labeled data.
BibTeX
@inproceedings{su2021self,
  author = {Yukun Su and Guosheng Lin and Qingyao Wu},
  title = {{Self-supervised 3D Skeleton Action Representation Learning with Motion Consistency and Continuity}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01308},
}
Figure