Non-rigid Point Cloud Registration with Neural Deformation Pyramid

Yang Li, Tatsuya Harada

Abstract

Non-rigid point cloud registration is a fundamental challenge in computer vision and graphics, complicated by the high degrees of freedom inherent in non-rigid motion. We present the Neural Deformation Pyramid (NDP), a hierarchical approach that decomposes complex deformations into a multi-level architecture. By utilizing a series of Multi-Layer Perceptrons (MLPs) with progressively increasing sinusoidal encoding frequencies, NDP effectively separates global rigid shifts from fine-grained local deformations. This architecture not only achieves state-of-the-art accuracy on benchmarks like 4DMatch and 4DLoMatch but also provides a 50-fold speedup over existing coordinate-MLP methods. Our approach demonstrates robust performance in both unsupervised and supervised settings, particularly in challenging partial-to-partial registration scenarios.

Introduction

Point cloud registration—the process of aligning two or more 3D datasets into a single coordinate system—is relatively straightforward when the transformation is rigid. However, non-rigid registration, where objects can bend, stretch, or deform (such as a person walking or a flag waving), remains a significant hurdle. The search space for such transformations is massive because every point can potentially move independently.

Traditional methods often struggle with high computational complexity and sensitivity to partial overlaps, where sensors only capture one side of a deforming object. Recent "coordinate-MLP" methods, which represent motion as a continuous function, have shown promise in accuracy but are often too slow for practical use. The Neural Deformation Pyramid (NDP) addresses these bottlenecks through a structured, frequency-based decomposition of motion.

Methodology: The Neural Deformation Pyramid

Hierarchical Motion Decomposition

The core innovation of NDP is the decomposition of non-rigid motion into a pyramid of levels. Instead of a single network attempting to learn the entire deformation, the task is distributed across multiple MLPs. Each level in the pyramid is responsible for predicting a "motion increment" that refines the alignment produced by the previous level.

Sinusoidal Encoding and Frequency Control

To control the scale of motion at each level, NDP employs sinusoidal positional encoding. The 3D coordinates are mapped to a higher-dimensional space using sine and cosine functions of varying frequencies.

  • Top Levels: Use low-frequency encodings, forcing the MLP to represent smooth, global motions. This acts as a natural regularizer, ensuring the initial alignment is essentially rigid.
  • Bottom Levels: Use high-frequency encodings, allowing the MLPs to capture high-frequency details and complex local deformations, such as surface wrinkles or joint articulations.

Top-Down Optimization

The optimization process follows a coarse-to-fine strategy. The system first solves for global motion at the top of the pyramid. Once the top level converges, its output is fixed, and the next level begins estimating the residual motion. This "divide and conquer" strategy makes the optimization landscape more manageable and significantly accelerates convergence.

Continuous Representation and Robustness

Because NDP uses MLPs to map coordinates to motion, it creates a continuous deformation field. This continuity allows the model to plausibly interpolate motion for regions that are missing in one of the point clouds (the partial-to-partial problem). Furthermore, the authors utilize the $L_1$ norm for the Chamfer distance loss, which proves more robust against outliers and non-overlapping regions than the standard $L_2$ norm.

Experimental Evaluation

The NDP framework was evaluated across several rigorous benchmarks:

  • 4DMatch & 4DLoMatch: These datasets, derived from DeformingThings4D, were specifically filtered to focus on complex non-rigid motions rather than simple rigid transformations.
  • DeepDeform: This dataset provides real-world RGB-D scans of dynamic objects, including humans and animals, testing the model's applicability to noisy, real-world data.
  • KITTI Scene Flow: Used to test robustness in large-scale outdoor environments with varying point densities and cluttered backgrounds.

Key Results

Efficiency and Speed

One of the most significant contributions of NDP is its efficiency. By breaking the problem into frequency-controlled levels, the model optimizes approximately 50 times faster than previous coordinate-MLP approaches like Nerfies or NSFP. This brings coordinate-based non-rigid registration much closer to practical utility.

State-of-the-Art Accuracy

NDP achieved leading results on the 4DMatch benchmark. In "no-learned" (unsupervised) settings, where the model has no prior knowledge of the specific shapes, it outperformed existing methods by a wide margin. When integrated with learned correspondences (the LNDP variant), it set new standards for the 4DLoMatch benchmark, which is characterized by extremely low overlap (15%–45%) between scans.

Limitations and Future Directions

While NDP represents a major step forward, certain challenges remain:

  • Topological Changes: The model assumes a continuous Euclidean space, making it difficult to handle "tearing" or changes in topology (e.g., hands separating after being joined).
  • Non-Isometric Motion: The unsupervised version relies on the assumption that motion is somewhat distance-preserving (isometric). It may struggle with extreme geometric stretching.
  • Real-Time Processing: Although 50x faster than predecessors, the optimization still requires several seconds per pair. Future work may involve implementing the architecture in high-performance frameworks like tiny-cuda-nn to reach real-time speeds for robotics and XR applications.

Conclusion

The Neural Deformation Pyramid demonstrates that hierarchical, frequency-based decomposition is a powerful tool for non-rigid registration. By balancing global structure with local detail, NDP provides a fast, accurate, and robust solution for aligning complex deforming 3D data.

Summary updated

Abstract

Non-rigid point cloud registration is a key component in many computer vision and computer graphics applications. The high complexity of the unknown non-rigid motion make this task a challenging problem. In this paper, we break down this problem via hierarchical motion decomposition. Our method called Neural Deformation Pyramid (NDP) represents non-rigid motion using a pyramid architecture. Each pyramid level, denoted by a Multi-Layer Perception (MLP), takes as input a sinusoidally encoded 3D point and outputs its motion increments from the previous level. The sinusoidal function starts with a low input frequency and gradually increases when the pyramid level goes down. This allows a multi-level rigid to nonrigid motion decomposition and also speeds up the solving by 50 times compared to the existing MLP-based approach. Our method achieves advanced partialto-partial non-rigid point cloud registration results on the 4DMatch/4DLoMatch benchmark under both no-learned and supervised settings.
BibTeX
@inproceedings{li2022non,
  author = {Yang Li and Tatsuya Harada},
  title = {{Non-rigid Point Cloud Registration with Neural Deformation Pyramid}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=7EBIMgLmF1},
}
Figure