Exhaustive Search of Correspondences between Multimodal Remote Sensing Images Using Convolutional Neural Network

Mykhail L. Uss, Benoît Vozel, Vladimir V. Lukin, Kacem Chehdi2022 · DOI

TL;DR

This paper introduces DLSMarea, a novel deep learning approach for finding all plausible correspondences between multimodal remote sensing images, even in challenging scenarios like repetitive textures. Unlike previous methods that focus on a single best match, DLSMarea's CNN-based similarity measure generates a map identifying multiple matches with subpixel accuracy, crucial for robust image registration.

What Problem Does This Paper Address?

Image registration, the process of aligning images from different sources or times, is fundamental for analyzing remote sensing data. A critical step is finding correspondences (matching points) between images. This is particularly challenging for multimodal registration (e.g., optical vs. radar images) where image characteristics differ significantly. Traditional methods often struggle with repetitive patterns or weak true matches, potentially leading to missed correspondences or incorrect alignments. Existing deep learning methods, while powerful, are typically designed to find only the single most likely match, neglecting other valid correspondences that could be vital for robust registration. This paper tackles the problem of performing an exhaustive search for all plausible correspondences within a defined search area, going beyond finding just the single best match.

Methods

The core of the proposed solution is a Convolutional Neural Network (CNN) architecture called DLSMarea. It builds upon a previous "point" similarity measure (DLSM) but is adapted for an "area" search.

  • DLSMarea Architecture: The CNN takes a template image fragment (TI) and a reference image fragment (RI) as input. It uses a U-net architecture with shared weights to extract features from both images. A crucial "correlation layer" then computes correlation maps between these features. These maps are further processed by convolutional layers to output a similarity measure (SM) map covering the entire search zone.
  • Output SM Map: For each pixel in the output SM map, DLSMarea predicts:
  • The translation vector (Δx, Δy) to the best match.
  • The diagonal values of the covariance matrix (σx, σy) representing localization uncertainty.
  • A correlation coefficient (kxy).
  • Novel Loss Function: The key innovation lies in the training strategy. Since annotating all plausible false matches is infeasible, the authors propose a loss function with four terms:
  • Lmain.peak: Focuses on accurately localizing the true correspondence.
  • Ldiscrimination: Encourages the network to distinguish between true and false matches.
  • Lshift: Enforces translation invariance, ensuring that the SM map can be tiled without introducing artifacts. This term is critical for enabling the detection of multiple matches.
  • Lrotation: Helps characterize localization accuracy for anisotropic textures.
  • Multiple Correspondence Detection: After generating the SM map, a three-step algorithm (SMdet calculation, nonmaximum suppression, and subpixel refinement) is used to extract multiple correspondences, their subpixel locations, and their associated uncertainty.

Datasets

The paper utilizes existing multimodal remote sensing image datasets previously used for DLSM training. These datasets comprise pairs of registered images covering various modalities:

  • Visible-to-infrared
  • Optical-to-radar
  • Optical-to-DEM (Digital Elevation Model)
  • Radar-to-DEM

A total of 4,000,000 patch pairs were used for training and validation, and 60,000 patch pairs for testing. The authors note that these datasets are not newly introduced but are standard for this type of research.

Key Insights and Ideas

  • The Power of "Area" SM for Multiple Matches: The core insight is that by designing a CNN to output an SM map for an entire search area and enforcing translation invariance through the loss function, the network is naturally encouraged to learn and detect all plausible correspondences, not just the single best one.
  • Translation Invariance as a Multi-Match Enabler: The Lshift loss term, designed to ensure that SM maps can be tiled seamlessly, surprisingly becomes the key mechanism for detecting multiple matches. This is because it forces the network to learn consistent similarity patterns across overlapping regions, revealing secondary peaks in the SM map.
  • Handcrafted vs. Deep Learning for Exhaustive Search: While handcrafted methods like NCC and SIFT are inherently suited for exhaustive search (each point is calculated independently), they are outperformed by deep learning in terms of discriminative power. DLSMarea bridges this gap by offering deep learning's performance with an exhaustive search capability.
  • False Matches as Useful Information: The paper acknowledges that false matches, often seen as noise, can contain valuable information about image structures, similar to concepts in nonlocal filtering. DLSMarea leverages this by aiming to detect all plausible matches, including potentially useful false ones.

Key Results

  • Improved Discrimination: DLSMarea variants consistently outperform handcrafted methods (NCC, MI, SIFT-OCT, MIND) in terms of Area Under the Curve (AUC) for discriminating true from false matches, achieving AUC values up to 86.87% in the general case.
  • Comparable Localization Accuracy: DLSMarea demonstrates localization accuracy comparable to or better than the original "point" DLSM, localizing significantly more pairs with subpixel accuracy.
  • Effective Multi-Match Detection: Visualizations of SM maps show DLSMarea's ability to identify multiple distinct peaks, indicating the detection of multiple plausible correspondences, especially in areas with repetitive textures.
  • Computational Efficiency: While slower than some handcrafted methods, DLSMarea is computationally efficient for an "area" search, processing the entire search zone at once, which is advantageous for large images. The inference time decreases with larger search zones due to GPU parallelization.

Limitations and Future Work

  • Translation-Centric: The current DLSMarea is primarily designed for translation search and does not inherently support large rotation and scale errors. While small errors can be tolerated through data augmentation, significant geometric transformations would require pre-compensation.
  • Future Work: The authors suggest extending the method to handle large rotation and scale errors. They also highlight the potential application of DLSMarea for multimodal registration of complex images with repetitive structures, high noise, or large structural differences (e.g., optical-to-DEM registration). The paper also notes that further optimization of hyperparameters and CNN architecture is possible.

Summary updated

BibTeX
@misc{uss2022exhaustive,
  author = {Mykhail L. Uss and Benoît Vozel and Vladimir V. Lukin and Kacem Chehdi},
  title = {{Exhaustive Search of Correspondences between Multimodal Remote Sensing Images Using Convolutional Neural Network}},
  year = {2022},
  doi = {10.3390/s22031231},
}
Figure