GLA-Net: An Attention Network with Guided Loss for Mismatch Removal

Zhi Chen, Fan Yang, Wenbing TaoarXiv 2019 · arXiv

Abstract

Mismatch removal is a critical step in computer vision tasks such as 3D reconstruction and SLAM, where distinguishing correct feature matches (inliers) from outliers is essential. GLA-Net addresses the limitations of existing deep learning methods—namely class imbalance and context pollution—by introducing a Guided Loss function and an Inlier Attention Block. The Guided Loss directly optimizes for the F-score by dynamically adjusting loss weights, while the Inlier Attention Block filters out outlier noise during global context aggregation. Experimental results on benchmarks like St.Brown, WIDE, and COLMAP demonstrate that GLA-Net achieves state-of-the-art performance, significantly improving F1-scores through a robust coarse-to-fine architecture.

Introduction

In computer vision, finding correspondences between images is the foundation for spatial reasoning. However, initial "putative" matches are often contaminated by mismatches (outliers) caused by repetitive textures, lighting variations, or extreme viewpoint changes. While deep learning has framed this as a binary classification problem, two significant challenges persist:

  1. Class Imbalance: Outliers frequently outnumber inliers, leading standard loss functions like cross-entropy to produce models biased toward one class, sacrificing either precision or recall.
  2. Context Pollution: Traditional "Context Normalization" (CN) layers aggregate statistics across all matches. Because outliers are essentially noise, they pollute these global statistics, hindering the network's ability to isolate true inliers.

Methodology

GLA-Net (Guided Loss Attention Network) introduces two primary innovations to overcome these hurdles within a coarse-to-fine framework.

Guided Loss

Rather than relying on static weights for positive and negative classes, GLA-Net utilizes a mathematically derived Guided Loss. The authors established a direct link between the loss function and the $F_n$-score, a metric that balances precision and recall. During training, the algorithm calculates the derivatives of the $F_n$-score and dynamically adjusts loss weights ($\lambda$ and $\mu$). This ensures that minimizing the loss directly correlates with maximizing the $F_n$-score, guiding the model toward the optimal decision hyperplane.

Figure 3: The PRF curve diagram (a) illustrates how different loss weights affect precision, recall, and F1-score. Visual results (b) and (c) show the impact of different loss points on the final matching quality.

Inlier Attention Block (IA Block)

To solve the problem of context pollution, the Inlier Attention Block (IA Block) replaces standard Context Normalization. Unlike CN, which treats every match equally, the IA Block learns an "indicating matrix" that assigns higher weights to potential inliers and lower weights to outliers.

Figure 5: Comparison between (a) the standard Global Context Block and (b) the proposed Inlier Attention Block. The IAN uses an indicating matrix to filter out noise before calculating global statistics.

When calculating global context (mean and variance), the network effectively ignores the noise from outliers. This results in cleaner feature extraction and more robust classification.

Coarse-to-Fine Architecture

The overall architecture is split into two stages: a Crude Context Subnet that performs an initial pass to generate a preliminary indicating matrix, and a Fine Optimization Subnet that refines the matches using the weighted context from the first stage, mimicking the iterative refinement found in traditional methods like RANSAC.

Experimental Results

GLA-Net was evaluated on three challenging benchmarks: St.Brown (low inlier ratio ~7.6%), WIDE (viewpoint changes), and COLMAP (complex rotations).

The model consistently outperformed previous state-of-the-art methods such as LFGC-Net and NM-Net. On the St.Brown dataset, GLA-Net achieved an F1-score of 59.95%, a significant improvement over LFGC-Net's 49.83%. On the WIDE dataset, it reached a near-perfect 95.21% F1-score. Analysis of the IA Block revealed that by the final layers, the network assigned weights to inliers that were 20-25 times higher than those assigned to outliers, confirming the effectiveness of the attention mechanism in "silencing" noise.

Conclusion

GLA-Net demonstrates that mismatch removal can be significantly improved by aligning the training objective with the evaluation metric and using attention as a robust statistical filter. By preventing outlier pollution and directly optimizing for the F-score, the network finds a superior balance between precision and recall. Future work may focus on reducing the computational overhead of the attention blocks and exploring integration into real-time SLAM systems for mobile applications.

Summary updated

Abstract

Mismatch removal is a critical prerequisite in many feature-based tasks. Recent attempts cast the mismatch removal task as a binary classification problem and solve it through deep learning based methods. In these methods, the imbalance between positive and negative classes is important, which affects network performance, i.e., Fn-score. To establish the link between Fn-score and loss, we propose to guide the loss with the Fn-score directly. We theoretically demonstrate the direct link between our Guided Loss and Fn-score during training. Moreover, we discover that outliers often impair global context in mismatch removal networks. To address this issue, we introduce the attention mechanism to mismatch removal task and propose a novel Inlier Attention Block (IA Block). To evaluate the effectiveness of our loss and IA Block, we design an end-to-end network for mismatch removal, called GLA-Net \footnote{Our code will be available in Github later.}. Experiments have shown that our network achieves the state-of-the-art performance on benchmark datasets.
BibTeX
@misc{chen2019gla,
  author = {Zhi Chen and Fan Yang and Wenbing Tao},
  title = {{GLA-Net: An Attention Network with Guided Loss for Mismatch Removal}},
  year = {2019},
  eprint = {1909.13092},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.13092},
}
Figure