Partial Point Cloud Registration with Multi-view 2D Image Learning

Yue Zhang, Yue Wu, Wenping Ma, Maoguo Gong, Hao Li, Biao Hou

Abstract

Point cloud registration is a fundamental task in robotics and autonomous systems, yet it remains challenging when scans have low overlap or significant noise. This paper introduces IAPReg (Image-Assisted Partial Registration), a framework that enhances 3D registration by incorporating 2D semantic knowledge. By projecting 3D points into multi-view depth maps and utilizing pre-trained 2D vision models, IAPReg "hallucinates" semantic context without requiring actual RGB images. Experimental results on ModelNet40 and 3DMatch demonstrate that IAPReg achieves state-of-the-art performance, particularly in partial registration scenarios.

Introduction

Finding the spatial transformation that aligns two 3D scans—point cloud registration—is essential for building 3D maps and orienting robots. In real-world applications, sensors often capture only partial views of an object, leading to low-overlap scans that are difficult to align using purely geometric features. While humans use semantic understanding to orient objects, most 3D registration algorithms lack this global context. IAPReg bridges this gap by leveraging the robust semantic features learned by 2D Vision Transformers (ViT) to guide 3D alignment.

Methodology

The IAPReg framework is built upon three primary technical components designed to integrate 2D semantic intelligence into the 3D registration pipeline.

The IAPReg pipeline overview, illustrating the flow from 3D point clouds to multi-view projection, dual-modality encoding, and final weighted registration.

1. Multi-view 2D Projection

To utilize 2D vision models without RGB data, IAPReg projects 3D point clouds from six orthogonal viewpoints (front, back, left, right, top, and bottom) to generate synthetic depth maps. This process allows the model to interpret the 3D structure through the lens of 2D image processing.

2. Inter-Modality Multi-view Learning

The system employs a dual-encoder architecture: a 3D encoder (KPConv) for geometric features and a pre-trained 2D ViT for semantic features extracted from the depth maps. These features are fused using an "Inter-modality KNN" module, which enriches 3D points with global semantic context.

Detailed view of the Inter-Modality Multi-View Learning module, showing the self-attention and KNN-based fusion of 2D and 3D features.

3. 2D-Weighted SVD

Standard registration often fails due to outliers. IAPReg addresses this by using the fused 2D-3D features to calculate confidence weights. By comparing semantic similarity, the model can down-weight geometrically close but semantically inconsistent matches (e.g., matching a chair leg to a table leg), leading to a more robust Singular Value Decomposition (SVD) for the final transformation.

Experimental Results

IAPReg was evaluated on several benchmarks, showing significant improvements over existing methods:

  • ModelNet40: The framework achieved a 96.58% Registration Recall (RR), setting a new state-of-the-art.
  • ModelLoNet: In challenging low-overlap scenarios, RR reached 89.33%, a nearly 3% improvement over previous best results.
  • Real-world Scenarios: On the 3DMatch dataset, IAPReg outperformed all methods that do not use additional RGB data, demonstrating its effectiveness in handling sensor noise and complex indoor environments.

Ablation studies confirmed that the 2D semantic weighting is critical, as its removal led to a 1.6% drop in registration recall.

Conclusion

IAPReg demonstrates that pre-trained 2D vision models can serve as powerful semantic priors for 3D tasks, even in the absence of color information. While the multi-view projection and ViT encoding introduce some computational overhead, the gains in registration accuracy—especially for partial scans—are substantial. Future work will focus on optimizing the integration of these modalities and extending the framework to large-scale outdoor environments.

Summary updated

Abstract

Learning representations from numerous 2D image data has shown promising performance, yet very few works apply this representations to point cloud registration. In this paper, we explore how to leverage the 2D information to assist the point cloud registration, and propose IAPReg, an Image-Assisted Partial 3D point cloud Registration framework with the multi-view images generated by the input point cloud. It is expected to enrich 3D information with 2D knowledge, and leverage 2D knowledge to assist with point cloud registration. Specifically, we create multi-view depth maps by projecting the input point cloud from several specific views, and then extract 2D and 3D features using some well-established models. To fuse the information learned from 2D and 3D modalities, inter-modality multi-view learning module is proposed to enhance geometric information and complement semantic information. Weighted SVD is a common method to reduce the impact of inaccurate correspondences on registration. However, determining the correspondence weights is not trivial. Therefore, we design a 2D-weighted SVD method, where the 2D knowledge is employed to provide weight information of correspondences. Extensive experiments perform that our method outperform the state-of-the-art method without additional 2D training data.
BibTeX
@inproceedings{zhang2025partial,
  author = {Yue Zhang and Yue Wu and Wenping Ma and Maoguo Gong and Hao Li and Biao Hou},
  title = {{Partial Point Cloud Registration with Multi-view 2D Image Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i10.33121},
}
Figure