Introduction
Image-to-Point Cloud (I2P) registration is a fundamental task in computer vision, essential for visual localization, augmented reality, and autonomous navigation. It involves determining the 6-degree-of-freedom (6-DoF) camera pose by matching pixels in a 2D image to points in a 3D point cloud. Despite recent advances, existing deep learning models face two critical hurdles: poor generalization to unseen "open-domain" environments and extreme computational costs associated with dense pixel-to-point matching.
The Challenge of Open-Domain Registration
Most current I2P registration models overfit to the specific textures and geometries of their training datasets. When deployed in a new environment—an "open-domain" scenario—their performance often degrades significantly. Furthermore, achieving high-precision registration typically requires pixel-level cross-attention. For a standard image and a 100k-point cloud, this process generates a matching matrix with approximately $10^{10}$ elements, exceeding the memory and processing capabilities of modern GPUs.
Methodology: Leveraging Topology
To address these issues, Top-I2P introduces a framework that prioritizes scene topology over raw visual features. Topology—the spatial relationships and intersections between patches—remains more consistent across different environments than textures like color or lighting.
The Top-I2P pipeline consists of three core stages:
- Topology Graph Construction: Utilizing the Segment Anything Model (SAM), the system partitions both the 2D image and 3D point cloud into semantic patches. These patches serve as nodes in topology graphs ($G_I$ and $G_P$).
- Correction-based Topology Reasoning (CTR): To bridge the modality gap between 2D and 3D data, the model uses Graph Convolutional Networks (GCNs) and a Sinkhorn layer to predict overlapping patches. This identifies sparse, meaningful connections between the two spaces.
- Fast Topology Feature Interaction (FTI): By performing patch-to-patch interaction rather than dense pixel-to-point matching, Top-I2P reduces the computational burden by a factor of $10^6$. Information is only exchanged between topologically related patches, ensuring both efficiency and accuracy.
Experimental Results
The authors evaluated Top-I2P on several benchmarks, including 7-Scenes, ScanNet, and a self-collected real-world dataset. The model was specifically tested in "open-domain" settings, where it was trained on one scene (e.g., a kitchen) and tested on entirely different ones (e.g., stairs or offices).
Top-I2P demonstrates superior performance compared to state-of-the-art (SOTA) methods like 2D3D-MATR and FreeReg. On the ScanNet dataset, Top-I2P achieved a Registration Recall (RR) of 0.901, significantly higher than FreeReg (0.780). Beyond accuracy, the model is highly efficient, running at approximately 5.6 FPS with manageable memory overhead, making it viable for real-time applications.
Conclusion
Top-I2P represents a significant step forward in robust I2P registration. By focusing on topology as a scene-independent cue and leveraging foundational models like SAM, it overcomes the generalization and efficiency barriers of previous methods. While challenges remain in textureless or extremely low-resolution environments, the shift toward topological reasoning provides a more reliable foundation for downstream tasks in general scenes.
Summary updated