A Survey on the Deep Learning-Based Mismatch Removal: Principles and Methods

Shiyu Chen, Cailong Deng, Yong Zhang, Yong Wang, Qixin Zhang, Zhimin ZhouIEEE Access 2023 · DOI

Abstract

Due to the inherent limitations of matching algorithms and the complexities associated with image contents, mismatches are inevitable and can have detrimental effects on downstream tasks in computer vision and remote sensing. This paper presents a comprehensive survey of deep learning (DL)-based mismatch removal principles and methods. We provide a summary of network architectures, techniques for extracting geometrical information, and various training modes. Specifically, we highlight the importance of permutation invariance in mining operations, enumerate a majority of existing mining methods, and provide an explanation of their permutation invariant properties. Furthermore, we present both the intuitive motivation and mathematical analysis of commonly used methods, elucidating their underlying principles and efficacy. Finally, we predict upcoming trends based on the findings of our review, aiming to provide valuable insights into mismatch removal techniques and guide their practical applications.

Introduction: The Challenge of Mismatches

Image matching is a cornerstone of computer vision, essential for tasks like 3D reconstruction, autonomous navigation, and remote sensing. However, the initial matching process is often messy, producing a high volume of "mismatches" or outliers alongside correct pairs. These false positives arise from changes in lighting, perspective, or repetitive patterns in the scene.

Initial image matching often results in a high volume of false positives or mismatches, indicated by the red lines.

Traditional robust estimators like RANSAC have been the gold standard for decades, but they struggle when the outlier rate is extremely high (e.g., >90%) or when the geometric relationship is complex and non-rigid. While many DL-based filters have been proposed recently, they were often "hidden" as sub-components of larger matching pipelines. This survey pulls them into the spotlight, offering a unified taxonomy to understand how these neural networks learn to distinguish a good match from a bad one.

Methodology: Core Components of DL-Based Filters

The survey categorizes DL-based mismatch removal into three fundamental components:

1. Network Architectures

Most models follow one of two structural paradigms:

  • Linear Structure: Cascaded blocks, often utilizing residual connections to facilitate training in deeper networks.
  • "T" Structure: These architectures fuse features from multiple scales. By treating intermediate features equally, they prevent the loss of valuable geometric cues that might otherwise vanish in deeper layers of a purely linear stack.

2. Information Mining Blocks (IMBs)

IMBs are the functional core of these networks. Because a set of matches is essentially an unordered point cloud, these blocks must be permutation invariant—the output must remain consistent regardless of the input order. Key techniques include:

  • Point-wise MLPs: Projecting 4D coordinates into high-dimensional feature spaces.
  • Context Normalization (CN): Acting as a global denoiser by normalizing features based on the mean and variance of the entire set. CN effectively highlights inliers by treating outliers as gross errors in a distribution.
  • Attention Mechanisms: Spatial Attention captures "motion coherence" (the principle that neighboring correct matches should move together), while Channel Attention suppresses noisy feature dimensions.
  • Graph Neural Networks (GNNs): Treating matches as nodes in a graph to explicitly model local and global geometric relationships.

3. Training Modes

The paper distinguishes between Supervised Learning, which relies on ground-truth labels or epipolar geometry constraints, and Unsupervised Learning, which utilizes Reinforcement Learning or weighted regression to maximize "consensus" without requiring explicit labels.

Key Insights and Principles

A critical "aha" moment in the survey is the mathematical link between modern deep learning techniques and traditional statistics. For instance, Context Normalization is shown to be a neural equivalent to traditional statistical "data-snooping."

Furthermore, the requirement for permutation invariance explains why standard CNNs are unsuitable for this task. Operations like $1 \times 1$ convolutions (Point-wise MLPs) and global pooling are the necessary building blocks for processing unordered sets of correspondences. In unsupervised frameworks, mismatch removal is often framed as a "game" where the agent is rewarded for finding the largest set of points that agree on a single geometric model.

Evolution and Performance

The survey tracks the evolution of the field from the seminal PointNet (2017) to state-of-the-art 2023 models like JRA-Net. Early methods relied on simple MLPs and Normalization, whereas modern SOTA methods almost universally incorporate Attention and GNNs to capture complex, non-rigid relationships.

Modern deep learning methods effectively filter outliers, preserving a high density of correct inlier matches (green lines).

DL-based methods now routinely handle outlier rates exceeding 80-90%, significantly outperforming traditional handcrafted methods in both speed and precision across standard benchmarks like YFCC100M (outdoor) and SUN3D (indoor).

Future Directions

The authors identify three major frontiers for future research:

  1. Multiple Representations: Current models often assume a single global geometric transformation. Future work must address scenes with multiple moving objects or sudden depth changes (multi-model consensus).
  2. The Labeling Bottleneck: To overcome the cost of supervised learning, there is a push toward robust unsupervised frameworks that learn directly from raw image pairs.
  3. Few-Shot Learning: Developing methods that can generalize to new sensor types or environments with minimal training data is critical for practical applications like drone navigation and rapid mapping.

Summary updated

Abstract

Due to the inherent limitations of matching algorithms and the complexities associated with image contents, mismatches are inevitable and can have detrimental effects on downstream tasks in computer vision and remote sensing. Researchers have published numerous reviews on mismatch removal, which may suffer from two primary deficiencies. Firstly, these reviews are often embedded within studies that primarily focus on image matching, thereby limiting the detailed and comprehensive analysis of mismatch removal methods. Secondly, reviews of deep learning (DL)-based methods, despite their numerous existence and interconnection, tend to be fragmentary and lack a systematic approach. To address these two shortcomings, this paper presents a comprehensive survey of DL-based mismatch removal principles and methods. We provide a summary of network architectures, techniques for extracting geometrical information, and various training modes. Specifically, we highlight the importance of permutation invariance in mining operations, enumerate a majority of existing mining methods, and provide an explanation of their permutation invariant properties. Furthermore, we present both the intuitive motivation and mathematical analysis of commonly used methods, elucidating their underlying principles and efficacy. In the conclusion, we predict upcoming trends based on the findings of our review, aiming to provide valuable insights into mismatch removal techniques and guide their practical applications.
BibTeX
@inproceedings{chen2023survey,
  author = {Shiyu Chen and Cailong Deng and Yong Zhang and Yong Wang and Qixin Zhang and Zhimin Zhou},
  title = {{A Survey on the Deep Learning-Based Mismatch Removal: Principles and Methods}},
  booktitle = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2023.3320682},
}
Figure