Abstract
Point cloud registration is a fundamental task in robotics and autonomous systems, yet it remains challenging when scans have low overlap or significant noise. This paper introduces IAPReg (Image-Assisted Partial Registration), a framework that enhances 3D registration by incorporating 2D semantic knowledge. By projecting 3D points into multi-view depth maps and utilizing pre-trained 2D vision models, IAPReg "hallucinates" semantic context without requiring actual RGB images. Experimental results on ModelNet40 and 3DMatch demonstrate that IAPReg achieves state-of-the-art performance, particularly in partial registration scenarios.
Introduction
Finding the spatial transformation that aligns two 3D scans—point cloud registration—is essential for building 3D maps and orienting robots. In real-world applications, sensors often capture only partial views of an object, leading to low-overlap scans that are difficult to align using purely geometric features. While humans use semantic understanding to orient objects, most 3D registration algorithms lack this global context. IAPReg bridges this gap by leveraging the robust semantic features learned by 2D Vision Transformers (ViT) to guide 3D alignment.
Methodology
The IAPReg framework is built upon three primary technical components designed to integrate 2D semantic intelligence into the 3D registration pipeline.
1. Multi-view 2D Projection
To utilize 2D vision models without RGB data, IAPReg projects 3D point clouds from six orthogonal viewpoints (front, back, left, right, top, and bottom) to generate synthetic depth maps. This process allows the model to interpret the 3D structure through the lens of 2D image processing.
2. Inter-Modality Multi-view Learning
The system employs a dual-encoder architecture: a 3D encoder (KPConv) for geometric features and a pre-trained 2D ViT for semantic features extracted from the depth maps. These features are fused using an "Inter-modality KNN" module, which enriches 3D points with global semantic context.
3. 2D-Weighted SVD
Standard registration often fails due to outliers. IAPReg addresses this by using the fused 2D-3D features to calculate confidence weights. By comparing semantic similarity, the model can down-weight geometrically close but semantically inconsistent matches (e.g., matching a chair leg to a table leg), leading to a more robust Singular Value Decomposition (SVD) for the final transformation.
Experimental Results
IAPReg was evaluated on several benchmarks, showing significant improvements over existing methods:
- ModelNet40: The framework achieved a 96.58% Registration Recall (RR), setting a new state-of-the-art.
- ModelLoNet: In challenging low-overlap scenarios, RR reached 89.33%, a nearly 3% improvement over previous best results.
- Real-world Scenarios: On the 3DMatch dataset, IAPReg outperformed all methods that do not use additional RGB data, demonstrating its effectiveness in handling sensor noise and complex indoor environments.
Ablation studies confirmed that the 2D semantic weighting is critical, as its removal led to a 1.6% drop in registration recall.
Conclusion
IAPReg demonstrates that pre-trained 2D vision models can serve as powerful semantic priors for 3D tasks, even in the absence of color information. While the multi-view projection and ViT encoding introduce some computational overhead, the gains in registration accuracy—especially for partial scans—are substantial. Future work will focus on optimizing the integration of these modalities and extending the framework to large-scale outdoor environments.
Summary updated