Abstract
In this paper, we present a novel registration framework, HumanReg, that learns a non-rigid transformation between two human point clouds end-to-end. We introduce body prior into the registration process to efficiently handle this type of point cloud. Unlike most existing supervised registration techniques that require expensive point-wise flow annotations, HumanReg can be trained in a self-supervised manner benefiting from a set of novel loss functions. To make our model better converge on real-world data, we also propose a pretraining strategy, and a synthetic dataset (HumanSyn4D) consists of dynamic, sparse human point clouds and their auto-generated ground truth annotations. Our experiments show that HumanReg achieves state-of-the-art performance on the CAPE-512 dataset and gains a qualitative result on another more challenging real-world dataset. Furthermore, our ablation studies demonstrate the effectiveness of our synthetic dataset and novel loss functions.
Introduction
Capturing 3D data of moving humans often results in "point clouds"—collections of coordinates in space. However, data from real-world sensors like LiDAR is often sparse, noisy, and plagued by occlusions. To make sense of this data over time, researchers use non-rigid registration, which involves calculating "scene flow"—the 3D motion vector for every single point between two frames.
The challenge is twofold: humans bend and stretch, making the math significantly harder than for rigid objects, and most high-performing models require supervised learning with point-wise ground truth flow, which is nearly impossible to label manually for real-world data. HumanReg addresses these issues by introducing a body prior and a self-supervised training pipeline.
Methodology
The HumanReg architecture consists of a shared 3D ResUNet backbone that feeds into two specialized "heads":
- Segmentation Head: This predicts which body part (e.g., left arm, torso) each point belongs to. By introducing this "body prior," the model learns more meaningful features than if it were just looking at raw geometry.
- Correspondence Head: This uses a soft-matching mechanism to estimate the 3D flow between frames.
The "Part-Rigid" Innovation
The core of HumanReg is its self-supervised loss function. While the whole human body is non-rigid, individual segments (like a forearm) behave almost rigidly. The authors introduce a Part-Rigid Loss that forces the estimated flow of points within the same body part to fit a single rigid transformation. This acts as a powerful regularizer, helping the model ignore sensor noise and focus on plausible human motion.
Training Pipeline
The training involves Supervised Pretraining on a synthetic dataset to provide a "warm start," followed by Self-supervised Finetuning on unlabeled real-world data using Chamfer, smoothness, clustering, and the part-rigid losses.
Datasets
The paper utilizes three main datasets to validate the framework:
- HumanSyn4D (New): A synthetic dataset created in Unity simulating four LiDARs scanning human avatars. It provides perfect ground truth for both body parts and scene flow.
- CAPE-512: A modified version of the CAPE dataset downsampled to 512 points to simulate the extreme sparsity of long-range LiDAR.
- BasketballPlayer (New): A challenging real-world dataset collected using four Livox LiDARs during a basketball match, featuring fast movement and heavy occlusion.
Key Results and Insights
- State-of-the-Art Accuracy: On the CAPE-512 benchmark, HumanReg achieved a 3D End-Point Error (EPE3D) of 3.22 cm, a significant improvement over previous leaders like PointPWC (6.32 cm) and Synorim (6.62 cm).
- Robustness to Sparsity: While other self-supervised methods failed to converge when point counts dropped, HumanReg remained stable even with only 128 points per frame.
- Segmentation as a Regularizer: Forcing the model to learn body parts provides latent pose information that helps the correspondence head resolve ambiguities.
- Synthetic-to-Real Generalization: Pretraining on HumanSyn4D is essential for convergence on messy real-world LiDAR data distributions.
Conclusion
HumanReg demonstrates that treating the human body as a collection of rigid parts allows for effective registration even in sparse, noisy environments. By leveraging synthetic data for pretraining and a novel part-rigid loss for self-supervision, the framework achieves superior performance without the need for manual annotations. Future work may involve incorporating temporal context across multiple frames to further improve stability in long sequences.
Summary updated