Guiding Local Feature Matching with Surface Curvature

Shuzhe Wang, Juho Kannala, Marc Pollefeys, Daniel Barath

Abstract

Local feature matching is a fundamental task in computer vision, yet most state-of-the-art matchers rely exclusively on 2D image intensity and texture. This paper introduces the Curvature Similarity Extractor (CSE), a plug-in module designed to incorporate 3D surface geometry into the matching process. By estimating local surface curvature from monocular depth maps, the CSE provides a viewpoint-invariant geometric signature that helps reject inconsistent matches. When integrated with popular matchers like SuperGlue and LoFTR, the proposed method achieves state-of-the-art performance across multiple benchmarks.

Introduction

Finding corresponding physical points across different images is essential for 3D reconstruction (SfM) and autonomous navigation (SLAM). While deep learning has significantly improved matching robustness, standard methods often struggle with repetitive patterns or surfaces that appear similar in 2D but differ in 3D structure. For example, a flat wall and a flat floor might share similar textures, leading to false positives in a purely 2D matching framework.

The authors propose that surface curvature—an intrinsic property of the 3D world—can serve as a powerful filter. Because curvature is invariant to camera viewpoint, it allows the system to verify if two points belong to the same physical structure based on their local shape rather than just their appearance.

Methodology

The Curvature Similarity Extractor Pipeline

The core contribution of this work is the Curvature Similarity Extractor (CSE), which can be integrated into existing detector-based or detector-free matching architectures. The pipeline begins by predicting a monocular depth map from the input images using a pre-trained model like MiDaS.

The Curvature Similarity Extractor (CSE) pipeline integrates depth prediction and curvature similarity into existing transformer-based matchers.

Geometric Feature Extraction

Once the depth map is obtained, the system performs the following steps to extract geometric signatures:

  1. Quadric Fitting: For each feature point, the system identifies a small local patch in the 3D depth map. It then fits a quadric surface—specifically an ellipsoid—to this local point cloud.
  2. Curvature Extraction: From the fitted ellipsoid, the system calculates principal curvatures or radii along its axes. These values are inherently 3D and remain constant regardless of camera rotation or translation.
  3. Similarity Scoring: A similarity function compares the curvature of a point in the first image with candidate points in the second. Matches between points with significantly different geometric profiles are penalized.
The system fits quadric surfaces to local depth patches to extract principal curvatures, providing a geometric signature invariant to camera viewpoint.

Integration and Training

The resulting curvature similarity score is linearly combined with the matching probabilities generated by standard 2D matchers. Crucially, the entire pipeline is differentiable. This allows for end-to-end fine-tuning of the depth prediction network, encouraging it to produce depth maps that prioritize the geometric details most useful for matching rather than just visual accuracy.

Experimental Results

The authors evaluated the CSE module on several major benchmarks, including ScanNet (indoor), MegaDepth (outdoor), and the Aachen Day-Night localization challenge.

Key findings include:

  • Performance Gains: The addition of CSE consistently improved the Area Under the Curve (AUC) for pose estimation. On ScanNet, AUC@5° improved by 1.0–1.5%, while MegaDepth saw gains of up to 2.5%.
  • Robustness: The method proved particularly effective at eliminating incorrect matches in areas with repetitive textures but distinct geometric profiles.
  • Versatility: As a plug-and-play module, CSE successfully boosted the performance of various baseline matchers, including LoFTR and QuadTree.

Conclusion and Future Work

By leveraging 3D surface curvature, the CSE module addresses a significant blind spot in modern 2D feature matchers. While the approach introduces some computational overhead due to depth prediction and quadric fitting, it provides a robust geometric constraint that significantly improves matching accuracy. Future research may focus on optimizing the efficiency of the depth prediction stage and improving performance near object edges where depth discontinuities can complicate curvature estimation.

Summary updated

Abstract

We propose a new method, called curvature similarity extractor (CSE), for improving local feature matching across images. CSE calculates the curvature of the local 3D surface patch for each detected feature point in a viewpoint-invariant manner via fitting quadrics to predicted monocular depth maps. This curvature is then leveraged as an additional signal in feature matching with off-the-shelf matchers like SuperGlue and LoFTR. Additionally, CSE enables end-to-end joint training by connecting the matcher and depth predictor networks. Our experiments demonstrate on large-scale real-world datasets that CSE consistently improves the accuracy of state-of-the-art methods. Fine-tuning the depth prediction network further enhances the accuracy. The proposed approach achieves state-of-the-art results on the ScanNet dataset, showcasing the effectiveness of incorporating 3D geometric information into feature matching.1
BibTeX
@inproceedings{wang2023guiding,
  author = {Shuzhe Wang and Juho Kannala and Marc Pollefeys and Daniel Barath},
  title = {{Guiding Local Feature Matching with Surface Curvature}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01648},
}
Figure