ULMR: An Unsupervised Learning Framework for Mismatch Removal

Cailong Deng, Shiyu Chen, Yong Zhang, Qixin Zhang, Feiyan ChenSensors 2022 · DOI

TL;DR

This paper introduces ULMR, a novel unsupervised learning framework for removing incorrect matches (mismatches) between image pairs. By leveraging Deep Reinforcement Learning (DRL), ULMR trains a network to identify and discard mismatches without requiring any labeled data, outperforming existing methods in precision and stability.

What Problem Does This Paper Address?

Accurate matching of corresponding points between images is fundamental for many computer vision tasks like 3D reconstruction and change detection. However, variations in lighting, viewpoint, and image quality often lead to "mismatches" – incorrect point correspondences. Removing these mismatches is crucial for reliable downstream applications. While deep learning methods have shown promise, they typically require large amounts of labeled data, which is time-consuming and prone to errors. Existing unsupervised methods often have limitations, such as requiring prior knowledge of the outlier rate or struggling with unsmooth loss functions. This paper aims to bridge this gap by proposing an unsupervised approach that effectively handles mismatches.

Methods

The core of ULMR is its application of Deep Reinforcement Learning (DRL) to the mismatch removal problem. The authors frame the process as a "game" where:

  • State: The initial set of matched point pairs.
  • Policy: A classification network (e.g., NM-Net) that outputs a "relative matching probability" (RMP) for each point pair, indicating its likelihood of being a correct match.
  • Action: Sampling a subset of matched points based on the RMPs.
  • Reward: A score derived from the quality of a geometric model (e.g., fundamental matrix) estimated from the sampled subset. A larger consensus set (correct matches) leads to a higher reward.

ULMR uses a policy gradient approach to train the classification network. It iteratively samples subsets, calculates rewards, and updates the network's parameters to maximize the expected reward. Crucially, the scoring mechanism is unsupervised, relying on geometric consistency rather than ground truth labels. The framework is designed to be compatible with various permutation-invariant classification networks.

Datasets

The paper uses the following datasets:

  • Training Data: brown_bm_1 from the SUN3D database and st_peters_square from the Yahoo YFCC100M database. For supervised methods, these were processed to extract SIFT features, estimate poses, and label matches based on epipolar errors. For ULMR, only SIFT keypoints were used without explicit labeling.
  • Test Data: Eight pairs of real-scenario images, including indoor scenes with viewpoint changes and outdoor scenes with viewpoint and scale changes, captured by mobile phones and UAVs.
  • Application Experiments: The Reichstag dataset, containing 1174 image pairs with significant changes in viewpoint, illumination, and scale.

Key Insights and Ideas

  • DRL for Mismatch Removal: Framing mismatch removal as a DRL problem allows for an unsupervised learning approach, circumventing the need for labeled data.
  • Policy-Guided Sampling: Instead of random sampling (like RANSAC), ULMR uses a learned policy (RMPs from a classification network) to guide the sampling of point subsets, making the process more efficient and effective.
  • Unsupervised Reward Signal: The reward is derived from the geometric consistency of sampled points, providing a natural and unsupervised way to evaluate the quality of a subset.
  • Universality: The ULMR framework is designed to be compatible with various classification networks, offering flexibility and potential for integration with existing architectures.

Key Results

  • Testing Experiments: ULMR achieved the best precision (mean ~93.8%) and the smallest number of remaining false matches (#RFM) compared to RANSAC, GMS, LFGC, NM-Net, and ULCM. While not always having the most correct matches (#RCM), its results were sufficient for applications like self-localization.
  • Application Experiments: On the challenging Reichstag dataset, ULMR demonstrated superior stability (lowest box heights in MPA and MedMPA), accuracy (lowest mean and median MPA/MedMPA), and quality (lowest MaxPA) of remaining matches.
  • Ablation Studies:
    • Sampling Number: ULMR achieved better performance with significantly fewer samples (100) compared to RANSAC (1500), demonstrating faster training and inference.
  • Compatibility: Integrating ULMR with NM-Net, LFGC-Net, and PointNet all yielded excellent results, highlighting the framework's high compatibility. ULMR+NM-Net performed slightly better.

Limitations and Future Work

  • Geometric Model Dependency: The paper notes that ULMR's ability to score state-action pairs accurately relies on having a known and determined geometric model between image pairs. If this is not the case, expert demonstration methods might be needed, making it a supervised approach.
  • Future Work: The authors suggest that ULMR's potential for practical use is significant due to its performance, stability, and compatibility. The framework's ability to speed up training and its effectiveness with fewer samples are also highlighted as key advantages.

Summary updated

Abstract

Due to radiometric and geometric distortions between images, mismatches are inevitable. Thus, a mismatch removal process is required for improving matching accuracy. Although deep learning methods have been proved to outperform handcraft methods in specific scenarios, including image identification and point cloud classification, most learning methods are supervised and are susceptible to incorrect labeling, and labeling data is a time-consuming task. This paper takes advantage of deep reinforcement leaning (DRL) and proposes a framework named unsupervised learning for mismatch removal (ULMR). Resorting to DRL, ULMR firstly scores each state-action pair guided by the output of classification network; then, it calculates the policy gradient of the expected reward; finally, through maximizing the expected reward of state-action pairings, the optimal network can be obtained. Compared to supervised learning methods (e.g., NM-Net and LFGC), unsupervised learning methods (e.g., ULCM), and handcraft methods (e.g., RANSAC, GMS), ULMR can obtain higher precision, more remaining correct matches, and fewer remaining false matches in testing experiments. Moreover, ULMR shows greater stability, better accuracy, and higher quality in application experiments, demonstrating reduced sampling times and higher compatibility with other classification networks in ablation experiments, indicating its great potential for further use.
BibTeX
@inproceedings{deng2022ulmr,
  author = {Cailong Deng and Shiyu Chen and Yong Zhang and Qixin Zhang and Feiyan Chen},
  title = {{ULMR: An Unsupervised Learning Framework for Mismatch Removal}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s22166110},
}
Figure