Learning 2D-3D Correspondences To Solve The Blind Perspective-n-Point Problem

Liu Liu, Dylan Campbell, Hongdong Li, Dingfu Zhou, Xibin Song, Ruigang YangarXiv 2020 · OpenReview

Abstract

Conventional absolute camera pose via a Perspective-n-Point (PnP) solver often assumes that the correspondences between 2D image pixels and 3D points are given. When the correspondences between 2D and 3D points are not known a priori, the task becomes the much more challenging blind PnP problem. This paper proposes a deep CNN model which simultaneously solves for both the 6-DoF absolute camera pose and 2D--3D correspondences. Our model comprises three neural modules connected in sequence. First, a two-stream PointNet-inspired network is applied directly to both the 2D image keypoints and the 3D scene points in order to extract discriminative point-wise features harnessing both local and contextual information. Second, a global feature matching module is employed to estimate a matchability matrix among all 2D--3D pairs. Third, the obtained matchability matrix is fed into a classification module to disambiguate inlier matches. The entire network is trained end-to-end, followed by a robust model fitting (P3P-RANSAC) at test time only to recover the 6-DoF camera pose. Extensive tests on both real and simulated data have shown that our method substantially outperforms existing approaches, and is capable of processing thousands of points a second with the state-of-the-art accuracy.

Introduction

The Perspective-n-Point (PnP) problem is a fundamental task in computer vision, requiring the estimation of a camera's 6-DoF pose (rotation and translation) relative to a 3D environment. While standard PnP solvers rely on known 2D-3D correspondences, the "Blind PnP" problem arises when these matches are unknown. This creates a circular dependency: pose estimation requires matches, but finding matches typically requires an initial pose estimate. Traditional solutions are often either computationally expensive global solvers or local methods prone to local minima.

Proposed Architecture

The authors introduce an end-to-end deep learning framework that bypasses the need for initial pose estimates by learning to predict correspondences directly from geometric structures.

Overview of the proposed end-to-end pipeline for solving the blind PnP problem.

The pipeline consists of three primary stages:

  1. Two-Stream Feature Extraction: The network processes 2D and 3D point sets through separate branches. It utilizes a PointNet-inspired architecture that constructs K-Nearest Neighbor (KNN) graphs to capture local geometric features and applies "Context Normalization" to integrate global scene context.
  2. Global Feature Matching: Instead of simple nearest-neighbor matching, the model employs a Sinkhorn layer based on Optimal Mass Transport theory. This module processes a pairwise distance matrix to iteratively generate a "matchability matrix" that satisfies global constraints, effectively resolving local ambiguities.
The global feature matching module utilizing Sinkhorn's algorithm to produce prioritized matches.
  1. Inlier Classification: Putative matches are filtered through a classification module to distinguish true inliers from outliers.

At test time, the top-K prioritized matches are passed to a standard P3P-RANSAC estimator to compute the final 6-DoF camera pose.

Key Insights

A central finding of this work is that local geometric structures in 3D space often retain enough distinctiveness after 2D projection for deep networks to learn direct matching without traditional visual descriptors like SIFT.

Furthermore, the use of the Sinkhorn algorithm is pivotal. By treating matching as an optimal transport problem, the network avoids the false positives common in local-only matching methods. The authors also introduced an "inlier set probability maximization loss," which they found superior to standard triplet or reprojection losses for optimizing the quality of the correspondence set.

Experimental Evaluation

The model was rigorously tested across three datasets: ModelNet40 (synthetic CAD models), NYU-RGBD (simulated indoor scenes), and MegaDepth (real-world outdoor landmarks).

Performance and Speed

  • Efficiency: The method achieves a 100x speedup over traditional global solvers like GOSMA, processing thousands of points in approximately 0.15 seconds.
  • Accuracy: On the MegaDepth dataset, the proposed method achieved a median rotation error of 0.056°, significantly outperforming GOSMA (86.784°) and SoftPOSIT (21.39°) under similar time constraints.
  • Robustness: The system remains accurate even with outlier ratios as high as 70-80%.

Conclusion

This framework demonstrates that deep learning can effectively solve the blind PnP problem by learning geometric descriptors and global matching constraints. While the current system uses a traditional RANSAC step, future iterations could integrate differentiable PnP solvers to enable a fully end-to-end gradient flow for pose estimation.

Summary updated

Abstract

Conventional absolute camera pose via a Perspective-n-Point (PnP) solver often assumes that the correspondences between 2D image pixels and 3D points are given. When the correspondences between 2D and 3D points are not known a priori, the task becomes the much more challenging blind PnP problem. This paper proposes a deep CNN model which simultaneously solves for both the 6-DoF absolute camera pose and 2D--3D correspondences. Our model comprises three neural modules connected in sequence. First, a two-stream PointNet-inspired network is applied directly to both the 2D image keypoints and the 3D scene points in order to extract discriminative point-wise features harnessing both local and contextual information. Second, a global feature matching module is employed to estimate a matchability matrix among all 2D--3D pairs. Third, the obtained matchability matrix is fed into a classification module to disambiguate inlier matches. The entire network is trained end-to-end, followed by a robust model fitting (P3P-RANSAC) at test time only to recover the 6-DoF camera pose. Extensive tests on both real and simulated data have shown that our method substantially outperforms existing approaches, and is capable of processing thousands of points a second with the state-of-the-art accuracy.
BibTeX
@misc{liu2020learning,
  author = {Liu Liu and Dylan Campbell and Hongdong Li and Dingfu Zhou and Xibin Song and Ruigang Yang},
  title = {{Learning 2D-3D Correspondences To Solve The Blind Perspective-n-Point Problem}},
  year = {2020},
  eprint = {2003.06752},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.06752},
}
Figure