HumanReg: Self-supervised Non-rigid Registration of Human Point Cloud

Yifan Chen, Zhiyu Pan, Zhicheng Zhong, Wenxuan Guo, Jianjiang Feng, Jie Zhou

Abstract

In this paper, we present a novel registration framework, HumanReg, that learns a non-rigid transformation between two human point clouds end-to-end. We introduce body prior into the registration process to efficiently handle this type of point cloud. Unlike most existing supervised registration techniques that require expensive point-wise flow annotations, HumanReg can be trained in a self-supervised manner benefiting from a set of novel loss functions. To make our model better converge on real-world data, we also propose a pretraining strategy, and a synthetic dataset (HumanSyn4D) consists of dynamic, sparse human point clouds and their auto-generated ground truth annotations. Our experiments show that HumanReg achieves state-of-the-art performance on the CAPE-512 dataset and gains a qualitative result on another more challenging real-world dataset. Furthermore, our ablation studies demonstrate the effectiveness of our synthetic dataset and novel loss functions.

Introduction

Capturing 3D data of moving humans often results in "point clouds"—collections of coordinates in space. However, data from real-world sensors like LiDAR is often sparse, noisy, and plagued by occlusions. To make sense of this data over time, researchers use non-rigid registration, which involves calculating "scene flow"—the 3D motion vector for every single point between two frames.

The challenge is twofold: humans bend and stretch, making the math significantly harder than for rigid objects, and most high-performing models require supervised learning with point-wise ground truth flow, which is nearly impossible to label manually for real-world data. HumanReg addresses these issues by introducing a body prior and a self-supervised training pipeline.

Overview of the HumanReg framework showing the pipeline from input frames to segmentation and correspondence outputs, including the cross-dataset adaptation strategy.

Methodology

The HumanReg architecture consists of a shared 3D ResUNet backbone that feeds into two specialized "heads":

  • Segmentation Head: This predicts which body part (e.g., left arm, torso) each point belongs to. By introducing this "body prior," the model learns more meaningful features than if it were just looking at raw geometry.
  • Correspondence Head: This uses a soft-matching mechanism to estimate the 3D flow between frames.

The "Part-Rigid" Innovation

The core of HumanReg is its self-supervised loss function. While the whole human body is non-rigid, individual segments (like a forearm) behave almost rigidly. The authors introduce a Part-Rigid Loss that forces the estimated flow of points within the same body part to fit a single rigid transformation. This acts as a powerful regularizer, helping the model ignore sensor noise and focus on plausible human motion.

Detailed architecture of HumanReg, illustrating the 3D ResUNet backbone, the segmentation and correspondence heads, and the various loss functions used during training.

Training Pipeline

The training involves Supervised Pretraining on a synthetic dataset to provide a "warm start," followed by Self-supervised Finetuning on unlabeled real-world data using Chamfer, smoothness, clustering, and the part-rigid losses.

Datasets

The paper utilizes three main datasets to validate the framework:

  1. HumanSyn4D (New): A synthetic dataset created in Unity simulating four LiDARs scanning human avatars. It provides perfect ground truth for both body parts and scene flow.
  2. CAPE-512: A modified version of the CAPE dataset downsampled to 512 points to simulate the extreme sparsity of long-range LiDAR.
  3. BasketballPlayer (New): A challenging real-world dataset collected using four Livox LiDARs during a basketball match, featuring fast movement and heavy occlusion.

Key Results and Insights

  • State-of-the-Art Accuracy: On the CAPE-512 benchmark, HumanReg achieved a 3D End-Point Error (EPE3D) of 3.22 cm, a significant improvement over previous leaders like PointPWC (6.32 cm) and Synorim (6.62 cm).
  • Robustness to Sparsity: While other self-supervised methods failed to converge when point counts dropped, HumanReg remained stable even with only 128 points per frame.
  • Segmentation as a Regularizer: Forcing the model to learn body parts provides latent pose information that helps the correspondence head resolve ambiguities.
  • Synthetic-to-Real Generalization: Pretraining on HumanSyn4D is essential for convergence on messy real-world LiDAR data distributions.

Conclusion

HumanReg demonstrates that treating the human body as a collection of rigid parts allows for effective registration even in sparse, noisy environments. By leveraging synthetic data for pretraining and a novel part-rigid loss for self-supervision, the framework achieves superior performance without the need for manual annotations. Future work may involve incorporating temporal context across multiple frames to further improve stability in long sequences.

Summary updated

Abstract

In this paper, we present a novel registration framework, HumanReg, that learns a non-rigid transformation between two human point clouds end-to-end. We introduce body prior into the registration process to efficiently handle this type of point cloud. Unlike most exsisting supervised registration techniques that require expensive point-wise flow annotations, HumanReg can be trained in a self-supervised manner benefiting from a set of novel loss functions. To make our model better converge on real-world data, we also propose a pretraining strategy, and a synthetic dataset (HumanSyn4D) consists of dynamic, sparse human point clouds and their auto-generated ground truth annotations. Our experiments shows that HumanReg achieves state-of-the-art performance on CAPE-512 dataset and gains a qualitative result on another more challenging real-world dataset. Furthermore, our ablation studies demonstrate the effectiveness of our synthetic dataset and novel loss functions. Our code and synthetic dataset is available at https://github.com/chenyifanthu/HumanReg.
BibTeX
@inproceedings{chen2024humanreg,
  author = {Yifan Chen and Zhiyu Pan and Zhicheng Zhong and Wenxuan Guo and Jianjiang Feng and Jie Zhou},
  title = {{HumanReg: Self-supervised Non-rigid Registration of Human Point Cloud}},
  booktitle = {3DV},
  year = {2024},
  doi = {10.1109/3dv62453.2024.00067},
}
Figure