Adaptive Assignment for Geometry Aware Local Feature Matching

Dihe Huang, Ying Chen, Shang Xu, Yong Liu, Wenlong Wu, Yikang Ding, Chengjie Wang, Fan Tang

TL;DR

AdaMatcher is a new detector-free feature matching framework that breaks the traditional "one-to-one" assignment bottleneck. By allowing flexible "many-to-one" matches and explicitly estimating scale differences between images, it achieves state-of-the-art performance in challenging scenarios involving extreme viewpoint and scale changes.

What Problem Does This Paper Address?

In computer vision, matching local features between two images is the bedrock of tasks like 3D reconstruction (SfM) and visual localization. Most modern methods rely on a "one-to-one" matching constraint (mutual nearest neighbor). While this simplifies the problem, it creates a fundamental geometric inconsistency when images have large scale or viewpoint differences.

Imagine a close-up photo and a far-away shot of the same building. A single large patch in the far-away shot might actually correspond to several distinct patches in the close-up. Forcing a one-to-one match here is physically incorrect and leads to "negative samples" during training that are actually true correspondences. This inconsistency, illustrated in the figure below, makes it difficult for models to converge and perform well in the wild.

Comparison of one-to-one vs. adaptive assignment. One-to-one forces a single match even when multiple points are geometrically consistent, leading to errors.

Methods

AdaMatcher introduces a detector-free pipeline that handles these geometric variations through three main components:

  1. Co-visible Feature Interaction (CFI): Instead of treating feature interaction and co-visibility estimation as separate tasks, AdaMatcher couples them. It uses a Transformer-based encoder-decoder structure where a global "co-visible query" helps the network focus on shared regions between images while suppressing distracting background noise.
  2. Adaptive Assignment: This is the core innovation. Instead of enforcing mutual nearest neighbors, the model performs a "one-way" assignment. By applying softmax operations independently across dimensions, it allows multiple patches in a high-resolution image to match a single patch in a lower-resolution image. This process also allows the model to explicitly calculate the relative scale ratio between the two images.
  3. Sub-pixel Refinement with Scale Alignment: Once coarse patch matches are found, the model uses the estimated scale ratio to upsample features from the smaller-scale image. This "scale alignment" ensures that the final sub-pixel refinement happens on features that are visually comparable, leading to much higher precision.
Figure 2. Architecture of AdaMatcher. A local feature CNN extracts two feature maps with size of 1/2 and 1/8 of the input image dimension. Afterwards, 1/8 size features of the two images are correlated by our feature interaction module, followed by an extra module to estimate the co-visible area shared between two images. Then the interacted patch features and co-visible probability maps are fed into an adaptive assignment and filtering module, yielding sufficient patch-level matches. Finally, proposal-matched features are sampled in the 1/2 size feature maps with a scale alignment to mitigate scale mismatch, followed by sub-pixel refinement.

Datasets

The authors trained and evaluated AdaMatcher using several standard benchmarks:

  • MegaDepth: Used for training and evaluating relative pose estimation in outdoor scenes. The authors specifically created a "scale-split" test set to measure performance across different scale ratios.
  • HPatches: Used to evaluate homography estimation under viewpoint and illumination changes.
  • Aachen Day-Night v1.1: A challenging dataset for visual localization under varying lighting conditions.
  • ScanNet & YFCC100M: Used to test the generalizability of the model for indoor and outdoor pose estimation.

Key Insights and Ideas

  • The One-to-One Bottleneck: The most significant "aha moment" is that the mutual nearest neighbor constraint is a self-imposed limitation. By relaxing this to an adaptive assignment, the model naturally handles the "many-to-one" nature of projective geometry.
  • Scale as a Byproduct: Relative scale isn't just a nuisance to be invariant to; it's a measurable property that can be derived directly from the matching distribution. Using this scale to align features before refinement is a powerful way to boost accuracy.
  • Co-visibility Guidance: Using co-visibility masks to filter matches during the interaction phase (rather than just as a post-processing step) helps the Transformer layers focus on relevant geometric structures.

Key Results

AdaMatcher consistently outperforms strong baselines like LoFTR and SuperGlue, particularly as the difficulty increases:

  • MegaDepth: At extreme scale ratios (4x or more), AdaMatcher-Quad achieves an AUC@5° of 26.56, more than doubling the performance of the original LoFTR (10.16).
  • HPatches: It achieves state-of-the-art homography accuracy, especially under viewpoint changes where it shows a substantial lead over detector-based methods.
  • Plug-and-Play: The authors demonstrated that their adaptive assignment and refinement modules could be added to SuperGlue, improving its AUC on MegaDepth across all scale ranges.
  • Efficiency: Despite the added complexity, the CFI module actually reduces computational costs by ~38% compared to the standard LoFTR interaction module by using a more efficient attention structure.

Limitations and Future Work

  • Inference Speed: While the overall pipeline (Matching + RANSAC) is faster due to a higher inlier ratio, the raw matching inference speed is slightly slower than LoFTR (157ms vs 105ms per pair).
  • Quantization Effects: In visual localization tasks (Aachen Day-Night), detector-free methods still slightly lag behind detector-based methods (like SuperPoint+SuperGlue) on day queries. The authors suggest this is because detector-free methods require more complex quantification during the database reconstruction process.
  • Future Directions: The authors believe the concept of adaptive assignment could be further integrated into multi-view geometry tasks and dense correspondence estimation.

Summary updated

Abstract

The detector-free feature matching approaches are currently attracting great attention thanks to their excellent performance. However, these methods still struggle at large-scale and viewpoint variations, due to the geometric inconsistency resulting from the application of the mutual nearest neighbour criterion (\ie, one-to-one assignment) in patch-level matching.Accordingly, we introduce AdaMatcher, which first accomplishes the feature correlation and co-visible area estimation through an elaborate feature interaction module, then performs adaptive assignment on patch-level matching while estimating the scales between images, and finally refines the co-visible matches through scale alignment and sub-pixel regression module.Extensive experiments show that AdaMatcher outperforms solid baselines and achieves state-of-the-art results on many downstream tasks. Additionally, the adaptive assignment and sub-pixel refinement module can be used as a refinement network for other matching methods, such as SuperGlue, to boost their performance further. The code will be publicly available at https://github.com/AbyssGaze/AdaMatcher.
BibTeX
@inproceedings{huang2022adaptive,
  author = {Dihe Huang and Ying Chen and Shang Xu and Yong Liu and Wenlong Wu and Yikang Ding and Chengjie Wang and Fan Tang},
  title = {{Adaptive Assignment for Geometry Aware Local Feature Matching}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure