Correspondence-Free Camera Pose Estimation by Matching Moments

Subhransu S. Bhattacharjee, Dylan Campbell, Rahul ShomearXiv 2026 · arXiv

Estimating where a camera is located relative to a 3D scene is a cornerstone of computer vision, essential for everything from augmented reality to autonomous driving. Usually, this requires knowing exactly which 2D pixel in an image corresponds to which 3D point in a model. But what if you don't have those matches?

TL;DR

MomentPnP is a learning-free algorithm that solves the "blind" camera pose estimation problem by matching statistical moments (shape descriptors) of point sets rather than individual points. It reduces rotation errors by up to 170x compared to state-of-the-art deep learning methods and, surprisingly, becomes more accurate as the number of points increases, scaling successfully to sets of one million points.

What Problem Does This Paper Address?

The authors tackle the Blind Perspective-n-Point (PnP) problem. In standard PnP, you are given a set of 2D image points and their corresponding 3D model points to calculate camera pose. In the "blind" version, you have the two sets of points, but you don't know which 2D point belongs to which 3D point.

This is a notoriously difficult combinatorial problem. If you have 1,000 points, there are $1000!$ possible pairings—a number far larger than the number of atoms in the universe. Existing solutions often rely on heavy pose searching (which is slow) or deep learning (which requires massive training data and often fails on new scenes). The authors ask: Can we skip the matching phase entirely and estimate the pose using the global "shape" of the point clouds?

Figure 1. Blind perspective-n-point with MomentPnP. The inputs are (a) an unordered set of 2D SIFT points [49] in a Sacr´e-Cœur image from the Phototourism dataset [37, 69], tracked by its structure-from-motion reconstruction [64], (b) the unordered 3D point set of those tracks, and known camera intrinsics K. The outputs are (c) the camera pose, with the matched model points and their rays in pink, and (d) the correspondences between projected model points (blue crosses) and observed 2D points (orange).

Methods

The core innovation is MomentPnP, a multi-step pipeline that treats point sets as probability distributions:

  1. Moment Matching: Instead of pairing points, the method calculates 14 statistical moments (up to the 4th order) for both the 2D observations and the projected 3D model. Moments like mean and covariance describe position and spread, while higher-order moments capture asymmetry and concentration.
  2. Fourier Features for Robustness: Standard polynomial moments are sensitive to outliers (a single point far away can ruin the average). To fix this, the authors use bounded Fourier features (sines and cosines), which limit the influence of any single outlier.
  3. Expectation-Maximization (EM) Refinement: Once an initial pose is found via moment matching, the algorithm uses EM to iteratively refine the pose and calculate "soft" correspondences.
  4. Alignment Check: Finally, the pose is verified using a statistical test to ensure the alignment didn't happen by sheer chance.

The authors provide a mathematical proof (Theorem 1) showing that these 14 moments are sufficient to uniquely identify the camera pose for almost any 3D model containing at least 20 points.

Higher-order moments allow the system to distinguish between different point distributions that might share the same mean and covariance.

Datasets

The method was evaluated on several diverse benchmarks:

  • MegaDepth: Real-world images with structure-from-motion reconstructions.
  • ModelNet40: Synthetic CAD models.
  • Data61: Panoramas paired with LiDAR scans.
  • ETH3D: High-density laser scans used to test scaling up to one million points.
  • Cambridge Landmarks, Phototourism, and 7-Scenes: Standard localization datasets.

Key Insights and Ideas

  • The Power of Permutation Invariance: Because moments are statistical averages, they are inherently "blind"—they don't care about the order of the points. This makes them a natural fit for correspondence-free problems.
  • Scaling is a Feature, Not a Bug: In most matching algorithms, more points mean more noise and harder computation. In MomentPnP, more points lead to more stable statistical averages. The authors showed that accuracy actually improves as point sets grow, even when individual correspondences become impossible to resolve.
  • Learning-Free Generality: Unlike deep learning baselines that are biased by their training data, MomentPnP is purely geometric. It works out-of-the-box on entirely new types of scenes (e.g., moving from a cathedral to a desktop) without retraining.

Key Results

MomentPnP significantly outperformed existing learned and search-based methods:

  • Accuracy: It reduced median rotation error by 5–170× compared to the best learned baselines across three major benchmarks.
  • Robustness: It maintained high accuracy even when 50% of the points were outliers (points that don't belong to the model).
  • Extreme Scaling: On dense ETH3D scans, it localized cameras with a median rotation error of just 0.00007° using one million points.
  • Efficiency: It typically finds a pose in under a second for benchmark-sized sets, though EM refinement on a million points can take several minutes.
Qualitative results showing MomentPnP (bottom row) accurately projecting 3D models onto images compared to other methods.

Limitations and Future Work

While powerful, the authors note a few hurdles:

  • Independent Selection: If the 2D points are detected independently of the 3D model (e.g., using SIFT on a query image vs. a pre-built map), the feature averages might not match perfectly even at the true pose.
  • Adversarial Outliers: While robust to uniform noise, structured or "adversarial" outliers can still mislead the moment matching.
  • Future Directions: The team suggests using Optimal Transport to better handle missing structures and extending the framework to estimate relative poses between two cameras or monocular depth.

Summary updated

Abstract

The blind perspective-$n$-point (PnP) problem is to estimate the pose of a calibrated camera from 2D and 3D point sets with unknown correspondences. We show that, for complete, noiseless observations, fourteen permutation-invariant image moments up to order four identify a fixed camera pose for almost every 3D point set containing at least twenty points. Our learning-free method, MomentPnP, matches these moments between the observed 2D points and the projected 3D points to initialize a camera pose without correspondences and refines the pose and correspondence weights using expectation--maximization. Across the test sets of three benchmarks, our method reduces median rotation error by $5$--$170\times$ relative to the best learned baseline, both with and without outliers. We recover accurate poses even with as many outliers as inliers. On dense ETH3D scans, pose accuracy improves as the point sets grow, even as individual correspondences become harder to resolve.
BibTeX
@misc{bhattacharjee2026correspondence,
  author = {Subhransu S. Bhattacharjee and Dylan Campbell and Rahul Shome},
  title = {{Correspondence-Free Camera Pose Estimation by Matching Moments}},
  year = {2026},
  eprint = {2611.20797},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20797},
}
Figure