Abstract
This paper proposes a novel concept to directly match feature descriptors extracted from 2D images with feature descriptors extracted from 3D point clouds. We use this concept to directly localize images in a 3D point cloud. We generate a dataset of matching 2D and 3D points and their corresponding feature descriptors, which is used to learn a Descriptor-Matcher classifier. To localize the pose of an image at test time, we extract keypoints and feature descriptors from the query image. The trained Descriptor-Matcher is then used to match the features from the image and the point cloud. The locations of the matched features are used in a robust pose estimation algorithm to predict the location and orientation of the query image. We carried out an extensive evaluation of the proposed method for indoor and outdoor scenarios and with different types of point clouds to verify the feasibility of our approach. Experimental results demonstrate that direct matching of feature descriptors from images and point clouds is not only a viable idea but can also be reliably used to estimate the 6-DOF poses of query cameras in any type of 3D point cloud in an unconstrained manner with high precision.
Introduction
To enable technologies like autonomous driving or augmented reality, a system needs to know its 6-DOF pose: exactly where it is in 3D space ($x, y, z$) and which way it is looking (roll, pitch, yaw). Traditionally, this has required building complex "Structure-from-Motion" (SfM) models from thousands of 2D images. While accurate, SfM is computationally expensive, fails in textureless areas, and struggles to scale to large environments.
Alternatively, Pose Regression Networks attempt to predict pose directly from images but often suffer from accuracy issues and overfitting. This paper addresses the gap by proposing a method to match standard 2D photos directly into dense 3D point clouds (from LIDAR or RGB-D sensors) without an intermediate SfM model.
Methodology
The core of the proposed approach is a novel Descriptor-Matcher pipeline that bridges the gap between 2D and 3D data domains.
Feature Extraction and Matching
The system extracts standard 2D features (SIFT) from images and 3D features (3D-SIFT and RIFT) from point clouds. Rather than using a simple distance metric, a two-stage classifier is employed:
- Coarse Stage: A Classification Tree quickly discards obvious mismatches.
- Fine Stage: A Random Forest provides high-precision verification for the remaining candidates.
Depth Block Filtering (DBF)
A key innovation is Depth Block Filtering, which ensures clean training data. When projecting 3D points onto a 2D plane, points behind occlusions (like walls) often project incorrectly. DBF uses a local depth-map strategy to filter these out, preventing the model from learning false correspondences.
Experimental Evaluation
The method was evaluated on several challenging benchmarks:
- Outdoor (Cambridge Landmarks): Urban datasets like King’s College and St. Mary Church, characterized by clutter and lighting changes.
- Indoor (Stanford RGB Localization): Kitchen and Living Room datasets captured with RGB-D sensors.
- Large-Scale Indoor (Baidu Indoor): A massive shopping mall scan (67 million points) with repetitive structures and significant occlusions.
Results and Discussion
The proposed method achieved state-of-the-art or highly competitive results across all test scenarios. On the King’s College dataset, it achieved a median positional error of just 8.7 cm and a rotational error of 0.147°, outperforming deep learning models like PoseNet.
In indoor environments, the precision was even higher, with errors as low as 1.17 cm in the Kitchen dataset. Even in the massive Baidu shopping mall, the system maintained a median error of 0.69 m, demonstrating superior scalability in feature-poor regions compared to traditional techniques.
Conclusion
Direct matching of feature descriptors from images and point clouds is a viable and reliable method for 6-DOF pose estimation. By operating directly on raw point clouds, the system remains independent of SfM models and can function in environments where traditional methods fail. Future work will explore replacing hand-crafted descriptors with learned deep-feature descriptors and improving computational efficiency through spatial indexing techniques like KD-trees.
Summary updated