MESAC: Learning to Remove Mismatches via Maximizing the Expected Score of Sample Consensuses

Shiyu Chen, Cailong Deng, Yong Zhang, Yong Wang, Qixin Zhang, Feiyan Chen, Zhimin ZhouIEEE Access 2023 · DOI

Abstract

Most learning-based methods require labelling the training data, which is time-consuming and gives rise to wrong labels. To address the labelling issues thoroughly, we propose an unsupervised learning framework to remove mismatches by maximizing the expected score of sample consensuses (MESAC). The proposed MESAC can train various permutation invariant networks (PINs) based on training data with no labels, and has three distinct merits: 1) the framework can train various PINs in an unsupervised mode such that these are immune to wrong labels; 2) the gradients of the expected score are explicitly calculated by a revised score-function estimator, which can avoid gradient explosion; 3) the distribution of matching probabilities is learned from the PIN and precisely modelled by a categorical distribution, which can decrease the sampling times and improve the computational efficiency accordingly. Experiments of testing datasets disclose that mean recall is increased by at most 77% when pure PINs are embedded in MESAC, and mean precision is also improved by 16%. Applications in pose recovery indicate that the success rates of MESAC-integrated PINs outperform the compared methods when training with neither matching labels nor ground truth epipolar geometry (EG) constraints, showing the great potential of MESAC in mismatch removal.

Introduction

In computer vision, establishing accurate correspondences between images is a prerequisite for tasks such as Structure-from-Motion (SfM), Simultaneous Localization and Mapping (SLAM), and autonomous navigation. However, initial matching sets often contain a high percentage of outliers (mismatches). Traditionally, robust estimators like RANSAC (Random Sample Consensus) have been used to filter these outliers by iteratively sampling minimal sets to find a geometric model with the highest consensus.

While deep learning has recently been applied to this problem, most existing methods rely on supervised learning. This requires large-scale datasets with ground-truth labels, which are expensive to obtain and often contain noise. Unsupervised methods offer a promising alternative but have historically struggled with numerical instability, specifically "gradient explosion," during the optimization of discrete sampling processes.

Methodology: The MESAC Framework

The core of the proposed approach is the Maximizing the Expected Score of Sample Consensuses (MESAC) framework. Unlike supervised methods that minimize a classification loss, MESAC treats the mismatch removal problem as an optimization of the expected RANSAC score.

Permutation Invariant Networks (PINs)

MESAC utilizes Permutation Invariant Networks (PINs) to process sets of matches. Because the order of matches in a set is arbitrary, the network must be invariant to permutations. The architecture typically involves Multi-Layer Perceptrons (MLPs) and local-global information mining modules to capture the relationships between matches.

Architecture of the Permutation Invariant Network (PIN) integrated within the MESAC framework.

As shown in the architecture above, the network takes a set of matches $\Psi$ and outputs a probability $p$ for each match. This probability is then used in two paths: one for calculating weights $w$ (via tanh+relu) for weighted Least Squares (LS) estimation, and another for determining inlier probabilities $r$ (via sigmoid).

Expected Score Maximization

The framework defines an objective function based on the expected score of a geometric model (e.g., Essential or Fundamental matrix) derived from sampled matches. By modeling the matching probabilities as a categorical distribution, the system can draw "minimal sets" of matches to estimate the geometry.

Overcoming Gradient Explosion

A significant contribution of this work is the use of a Revised Score-Function Estimator. To backpropagate through the discrete sampling step, the authors calculate gradients relative to the logarithm of the matching probabilities. This mathematical formulation stabilizes the training process, preventing the infinite gradients that often cause unsupervised models to fail. Additionally, a baseline subtraction technique is employed to reduce the variance of the gradient estimates, further smoothing the learning curve.

Experimental Evaluation

The researchers evaluated MESAC across four diverse datasets:

  1. MR3D: A new dataset of UAV imagery over a village.
  2. GL3D: A large-scale database of aerial and ground-level sequences.
  3. SUN3D: An indoor dataset featuring various room types.
  4. YFCC: A dataset of outdoor city scenes and landmarks.

Key Results

The integration of MESAC into existing PIN architectures yielded substantial performance improvements:

  • Recall and Precision: Mean recall increased by up to 77% (specifically for the DFENet architecture), while mean precision improved by 16%.
  • Pose Recovery: In camera pose estimation tasks, MESAC-trained models outperformed supervised counterparts. Notably, these models achieved superior results without ever seeing a single manual label or ground-truth epipolar geometry constraint during training.

Conclusion

MESAC represents a significant step forward in unsupervised mismatch removal. By framing the problem as the maximization of an expected consensus score and solving the stability issues associated with discrete sampling, the authors have created a framework that is both robust and "label-immune." Future work will focus on incorporating spatial dependencies between matches and exploring variational inference to further accelerate the training process.

Summary updated

Abstract

Most learning-based methods require labelling the training data, which is time-consuming and gives rise to wrong labels. To address the labelling issues thoroughly, we propose an unsupervised learning framework to remove mismatches by maximizing the expected score of sample consensuses (MESAC). The proposed MESAC can train various permutation invariant networks (PINs) based on training data with no labels, and has three distinct merits: 1) the framework can train various PINs in an unsupervised mode such that these are immune to wrong labels; 2) the gradients of the expected score are explicitly calculated by a revised score-function estimator, which can avoid gradient explosion; 3) the distribution of matching probabilities is learned from the PIN and precisely modelled by a categorical distribution, which can decrease the sampling times and improve the computational efficiency accordingly. Experiments of testing datasets disclose that mean recall is increased by at most 77% when pure PINs are embedded in MESAC, and mean precision is also improved by 16%. Applications in pose recovery indicate that the success rates of MESAC-integrated PINs outperform the compared methods when training with neither matching labels nor ground truth epipolar geometry (EG) constraints, showing the great potential of MESAC in mismatch removal.
BibTeX
@inproceedings{chen2023mesac,
  author = {Shiyu Chen and Cailong Deng and Yong Zhang and Yong Wang and Qixin Zhang and Feiyan Chen and Zhimin Zhou},
  title = {{MESAC: Learning to Remove Mismatches via Maximizing the Expected Score of Sample Consensuses}},
  booktitle = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2023.3280824},
}
Figure