Abstract
This paper introduces a novel self-supervised framework for learning 4D spatio-temporal features from dynamic point clouds. By leveraging the inherent temporal consistency of unlabelled point cloud sequences, the proposed method trains a model to predict the chronological order of shuffled clips. This "pre-text" task encourages the network to capture complex geometric and motion dynamics. Evaluated on action recognition and 3D sequence retrieval, the framework demonstrates significant performance gains, rivaling supervised methods and showcasing the potential of self-supervised learning for dynamic 3D perception.
Introduction
The field of 3D scene understanding has seen rapid progress, yet most state-of-the-art models remain heavily dependent on large-scale, human-annotated datasets. For 4D data—sequences of 3D point clouds over time—the annotation burden is particularly severe, as labeling every point across multiple frames is both time-consuming and expensive. To address this, researchers have turned to self-supervised learning to extract meaningful representations from raw, unlabelled data.
While previous self-supervised approaches have focused on static 3D objects or 2D video, this work targets dynamic 3D perception. The core idea is to use the "free" supervisory signal provided by the temporal flow of point cloud sequences to teach a neural network the underlying structure of human motion in 3D space.
Methods
The proposed self-supervised pipeline consists of three primary stages: Tuple Clip Sampling, Feature Extraction, and Recurrent Order Prediction.
- Tuple Clip Sampling (TCS): From a continuous point cloud sequence, several short, non-overlapping clips are extracted. Each clip contains multiple frames. These clips are then shuffled into a random permutation, creating a temporal "jigsaw puzzle."
- Feature Extraction (FE): Each shuffled clip is processed by a weight-shared 4D Convolutional Neural Network (CNN). The authors explored two specific architectures: 4D MinkNet, which utilizes sparse convolutions in 4D space, and MeteorNet, which aggregates neighborhood information in a spatio-temporal metric space.
- Recurrent Order Prediction (ROP): The resulting feature vectors are fed into a Long Short-Term Memory (LSTM) network. The LSTM is tasked with predicting the original chronological order from the $N!$ possible permutations. Solving this task requires the model to understand the physics and geometry of motion.
Datasets
The study utilizes two primary datasets:
- NTU-PCLs: A new, large-scale dataset created by the authors from the NTU-RGB+D benchmark. Using a quadratic registration model, they synchronized RGB and depth data to produce 56,880 4D point cloud sequences.
- MSRAction3D: A standard benchmark used to evaluate the transferability of the learned features to action recognition tasks.
Results and Discussion
The experimental results validate the effectiveness of the order prediction task. The 4D MinkNet achieved an accuracy of 85.3% on the pre-text task itself. More importantly, when the self-supervised weights were used to initialize a model for action recognition on MSRAction3D, performance improved from 75.62% (trained from scratch) to 86.31%. This significant boost demonstrates that the model successfully learned transferable motion features. Additionally, nearest neighbor retrieval experiments showed that the learned features are highly descriptive, allowing the model to identify similar actions without explicit category labels during pre-training.
Conclusion
This work presents a robust framework for self-supervised 4D feature learning. By predicting temporal order, models can learn complex motion dynamics from unlabelled data. While the approach currently depends on accurate point cloud registration and involves high computational costs for 4D convolutions, it opens new avenues for dynamic 3D perception. Future work may explore more granular signals, such as 3D point cloud flow, to further enhance the richness of the learned representations.
Summary updated