Unconstrained Matching of 2D and 3D Descriptors for 6-DOF Pose Estimation

Uzair Nadeem, Mohammed Bennamoun, Roberto Togneri, Ferdous SohelarXiv 2020 · OpenReview

Abstract

This paper proposes a novel concept to directly match feature descriptors extracted from 2D images with feature descriptors extracted from 3D point clouds. We use this concept to directly localize images in a 3D point cloud. We generate a dataset of matching 2D and 3D points and their corresponding feature descriptors, which is used to learn a Descriptor-Matcher classifier. To localize the pose of an image at test time, we extract keypoints and feature descriptors from the query image. The trained Descriptor-Matcher is then used to match the features from the image and the point cloud. The locations of the matched features are used in a robust pose estimation algorithm to predict the location and orientation of the query image. We carried out an extensive evaluation of the proposed method for indoor and outdoor scenarios and with different types of point clouds to verify the feasibility of our approach. Experimental results demonstrate that direct matching of feature descriptors from images and point clouds is not only a viable idea but can also be reliably used to estimate the 6-DOF poses of query cameras in any type of 3D point cloud in an unconstrained manner with high precision.

Introduction

To enable technologies like autonomous driving or augmented reality, a system needs to know its 6-DOF pose: exactly where it is in 3D space ($x, y, z$) and which way it is looking (roll, pitch, yaw). Traditionally, this has required building complex "Structure-from-Motion" (SfM) models from thousands of 2D images. While accurate, SfM is computationally expensive, fails in textureless areas, and struggles to scale to large environments.

Alternatively, Pose Regression Networks attempt to predict pose directly from images but often suffer from accuracy issues and overfitting. This paper addresses the gap by proposing a method to match standard 2D photos directly into dense 3D point clouds (from LIDAR or RGB-D sensors) without an intermediate SfM model.

Methodology

The core of the proposed approach is a novel Descriptor-Matcher pipeline that bridges the gap between 2D and 3D data domains.

The 2D-3D matching pipeline: from source point cloud and query image to final pose alignment.

Feature Extraction and Matching

The system extracts standard 2D features (SIFT) from images and 3D features (3D-SIFT and RIFT) from point clouds. Rather than using a simple distance metric, a two-stage classifier is employed:

  1. Coarse Stage: A Classification Tree quickly discards obvious mismatches.
  2. Fine Stage: A Random Forest provides high-precision verification for the remaining candidates.

Depth Block Filtering (DBF)

A key innovation is Depth Block Filtering, which ensures clean training data. When projecting 3D points onto a 2D plane, points behind occlusions (like walls) often project incorrectly. DBF uses a local depth-map strategy to filter these out, preventing the model from learning false correspondences.

Experimental Evaluation

The method was evaluated on several challenging benchmarks:

  • Outdoor (Cambridge Landmarks): Urban datasets like King’s College and St. Mary Church, characterized by clutter and lighting changes.
  • Indoor (Stanford RGB Localization): Kitchen and Living Room datasets captured with RGB-D sensors.
  • Large-Scale Indoor (Baidu Indoor): A massive shopping mall scan (67 million points) with repetitive structures and significant occlusions.

Results and Discussion

The proposed method achieved state-of-the-art or highly competitive results across all test scenarios. On the King’s College dataset, it achieved a median positional error of just 8.7 cm and a rotational error of 0.147°, outperforming deep learning models like PoseNet.

Detailed view of the 3D point cloud with projected 2D features used for robust pose estimation.

In indoor environments, the precision was even higher, with errors as low as 1.17 cm in the Kitchen dataset. Even in the massive Baidu shopping mall, the system maintained a median error of 0.69 m, demonstrating superior scalability in feature-poor regions compared to traditional techniques.

Conclusion

Direct matching of feature descriptors from images and point clouds is a viable and reliable method for 6-DOF pose estimation. By operating directly on raw point clouds, the system remains independent of SfM models and can function in environments where traditional methods fail. Future work will explore replacing hand-crafted descriptors with learned deep-feature descriptors and improving computational efficiency through spatial indexing techniques like KD-trees.

Summary updated

Abstract

This paper proposes a novel concept to directly match feature descriptors extracted from 2D images with feature descriptors extracted from 3D point clouds. We use this concept to directly localize images in a 3D point cloud. We generate a dataset of matching 2D and 3D points and their corresponding feature descriptors, which is used to learn a Descriptor-Matcher classifier. To localize the pose of an image at test time, we extract keypoints and feature descriptors from the query image. The trained Descriptor-Matcher is then used to match the features from the image and the point cloud. The locations of the matched features are used in a robust pose estimation algorithm to predict the location and orientation of the query image. We carried out an extensive evaluation of the proposed method for indoor and outdoor scenarios and with different types of point clouds to verify the feasibility of our approach. Experimental results demonstrate that direct matching of feature descriptors from images and point clouds is not only a viable idea but can also be reliably used to estimate the 6-DOF poses of query cameras in any type of 3D point cloud in an unconstrained manner with high precision.
BibTeX
@misc{nadeem2020unconstrained,
  author = {Uzair Nadeem and Mohammed Bennamoun and Roberto Togneri and Ferdous Sohel},
  title = {{Unconstrained Matching of 2D and 3D Descriptors for 6-DOF Pose Estimation}},
  year = {2020},
  eprint = {2005.14502},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.14502},
}

Loading related content...

Figure