Method Towards CVPR 2021 Image Matching Challenge

Xiaopeng Bi, Yu Chen, Xinyang Liu, Dehao Zhang, Ran Yan, Zheng Chai, Haotian Zhang, Xiao LiuarXiv 2021 · arXiv

Abstract

The Megvii-3D team presents their winning strategies for the CVPR 2021 Image Matching Challenge. Their approach centers on a robust pipeline combining DISK and SuperPoint features with SuperGlue matching. Key enhancements include aggressive multi-scale/multi-orientation feature extraction, semantic masking of dynamic objects, and an adaptive geometric verification strategy inspired by SLAM techniques.

Introduction

Image matching is a cornerstone of 3D reconstruction (SfM) and visual localization. However, real-world imagery often contains significant challenges such as extreme scale variations, rotations, and dynamic "noise" from pedestrians, vehicles, or the sky. Standard local features frequently struggle when viewpoint changes are drastic or when unreliable keypoints lead to false matches. This work proposes a pipeline designed to overcome these environmental hurdles.

Proposed Methodology

The team developed a multi-stage pipeline to ensure high repeatability and accuracy in challenging environments.

The multi-stage image matching pipeline proposed by the Megvii-3D team.

The pipeline consists of several key components:

  • Feature Extraction: Utilizing SuperPoint and DISK. SuperPoint features are refined via homographic adaptation to improve keypoint repeatability.
  • Pyramid Extraction: To achieve scale and rotation invariance, features are extracted across three scales and seven orientations.
  • Semantic Masking: Dynamic objects and unstable regions (e.g., sky) are masked using Mask-RCNN and PSPNet. A specialized classifier is used to distinguish between humans (to be masked) and sculptures (to be preserved).
  • Matching and Verification: SuperGlue is used for matching, supplemented by a "Guided Pyramid Matching" mode for difficult cases. Final matches are verified using DegenSAC and an Adaptive F/H strategy that selects the best geometric model (Fundamental or Homography) for the scene.

Key Results and Discussion

The integration of DISK + SuperGlue (Outdoor) proved to be the most effective configuration in the team's evaluations.

Comparison of Exact AUC scores for different feature and matcher combinations.

As shown in the table above, the DISK + SuperGlue combination achieved an AUC of 42.72 at 5°, a notable improvement over the official SuperPoint + SuperGlue baseline of 38.72. The multi-scale approach was particularly effective for the PragueParks dataset, where scale variation is a primary challenge.

Conclusion

The Megvii-3D team's approach demonstrates that while deep learning features are powerful, explicit handling of scale and rotation through pyramid extraction and classical geometric verification remains essential for state-of-the-art performance. Future work will address challenges related to extreme viewpoints (>3x scale difference) and non-rigid shape deformations.

Summary updated

Abstract

This report describes Megvii-3D team's approach towards CVPR 2021 Image Matching Workshop.
BibTeX
@misc{bi2021method,
  author = {Xiaopeng Bi and Yu Chen and Xinyang Liu and Dehao Zhang and Ran Yan and Zheng Chai and Haotian Zhang and Xiao Liu},
  title = {{Method Towards CVPR 2021 Image Matching Challenge}},
  year = {2021},
  eprint = {2108.04453},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.04453},
}
Figure