Self-supervised 4D Spatio-temporal Feature Learning via Order Prediction of Sequential Point Cloud Clips

Yang Liang, Haiyan Wang, Liang Yang, Xuejian Rong, Jinglun Feng, Yingli Tian

Abstract

This paper introduces a novel self-supervised framework for learning 4D spatio-temporal features from dynamic point clouds. By leveraging the inherent temporal consistency of unlabelled point cloud sequences, the proposed method trains a model to predict the chronological order of shuffled clips. This "pre-text" task encourages the network to capture complex geometric and motion dynamics. Evaluated on action recognition and 3D sequence retrieval, the framework demonstrates significant performance gains, rivaling supervised methods and showcasing the potential of self-supervised learning for dynamic 3D perception.

Introduction

The field of 3D scene understanding has seen rapid progress, yet most state-of-the-art models remain heavily dependent on large-scale, human-annotated datasets. For 4D data—sequences of 3D point clouds over time—the annotation burden is particularly severe, as labeling every point across multiple frames is both time-consuming and expensive. To address this, researchers have turned to self-supervised learning to extract meaningful representations from raw, unlabelled data.

Conceptual overview of the self-supervised 4D feature learning framework, where the model learns to predict the temporal order of shuffled point cloud clips.

While previous self-supervised approaches have focused on static 3D objects or 2D video, this work targets dynamic 3D perception. The core idea is to use the "free" supervisory signal provided by the temporal flow of point cloud sequences to teach a neural network the underlying structure of human motion in 3D space.

Methods

The proposed self-supervised pipeline consists of three primary stages: Tuple Clip Sampling, Feature Extraction, and Recurrent Order Prediction.

The complete system architecture, illustrating the sampling of raw RGBD sequences into clips, feature extraction via weight-shared 4D CNNs, and final order prediction using an LSTM.
  1. Tuple Clip Sampling (TCS): From a continuous point cloud sequence, several short, non-overlapping clips are extracted. Each clip contains multiple frames. These clips are then shuffled into a random permutation, creating a temporal "jigsaw puzzle."
  2. Feature Extraction (FE): Each shuffled clip is processed by a weight-shared 4D Convolutional Neural Network (CNN). The authors explored two specific architectures: 4D MinkNet, which utilizes sparse convolutions in 4D space, and MeteorNet, which aggregates neighborhood information in a spatio-temporal metric space.
  3. Recurrent Order Prediction (ROP): The resulting feature vectors are fed into a Long Short-Term Memory (LSTM) network. The LSTM is tasked with predicting the original chronological order from the $N!$ possible permutations. Solving this task requires the model to understand the physics and geometry of motion.

Datasets

The study utilizes two primary datasets:

  • NTU-PCLs: A new, large-scale dataset created by the authors from the NTU-RGB+D benchmark. Using a quadratic registration model, they synchronized RGB and depth data to produce 56,880 4D point cloud sequences.
  • MSRAction3D: A standard benchmark used to evaluate the transferability of the learned features to action recognition tasks.

Results and Discussion

The experimental results validate the effectiveness of the order prediction task. The 4D MinkNet achieved an accuracy of 85.3% on the pre-text task itself. More importantly, when the self-supervised weights were used to initialize a model for action recognition on MSRAction3D, performance improved from 75.62% (trained from scratch) to 86.31%. This significant boost demonstrates that the model successfully learned transferable motion features. Additionally, nearest neighbor retrieval experiments showed that the learned features are highly descriptive, allowing the model to identify similar actions without explicit category labels during pre-training.

Conclusion

This work presents a robust framework for self-supervised 4D feature learning. By predicting temporal order, models can learn complex motion dynamics from unlabelled data. While the approach currently depends on accurate point cloud registration and involves high computational costs for 4D convolutions, it opens new avenues for dynamic 3D perception. Future work may explore more granular signals, such as 3D point cloud flow, to further enhance the richness of the learned representations.

Summary updated

Abstract

Recently 3D scene understanding attracts attention for many applications, however, annotating a vast amount of 3D data for training is usually expensive and time consuming. To alleviate the needs of ground truth, we propose a self-supervised schema to learn 4D spatio-temporal features (i.e. 3 spatial dimensions plus 1 temporal dimension) from dynamic point cloud data by predicting the temporal order of sampled and shuffled point cloud clips. 3D sequential point cloud contains precious geometric and depth information to better recognize activities in 3D space compared to videos. To learn the 4D spatio-temporal features, we introduce 4D convolution neural networks to predict the temporal order on a self-created large scale dataset, NTU-PCLs, derived from the NTU-RGB+D dataset. The efficacy of the learned 4D spatio-temporal features is verified on two tasks: 1) Self-supervised 3D nearest neighbor retrieval; and 2) Self-supervised representation learning transferred for action recognition on a smaller 3D dataset. Our extensive experiments prove the effectiveness of the proposed self-supervised learning method which achieves comparable results w.r.t. the fully-supervised methods on action recognition on MSRAction3D dataset.
BibTeX
@inproceedings{liang2021self,
  author = {Yang Liang and Haiyan Wang and Liang Yang and Xuejian Rong and Jinglun Feng and Yingli Tian},
  title = {{Self-supervised 4D Spatio-temporal Feature Learning via Order Prediction of Sequential Point Cloud Clips}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00381},
}
Figure