Multimodal Remote Sensing Image Registration With Accuracy Estimation at Local and Global Scales

Mikhail L. Uss, Benoît Vozel, Vladimir V. Lukin, Kacem Chehdi

TL;DR

The RAE (Registration with Accuracy Estimation) method allows for the automatic alignment of vastly different remote sensing images—such as optical photos, radar maps, and digital elevation models—while providing a precise estimate of registration error for every pixel without needing any ground truth data. By treating potential accuracy as an input rather than just a result, the system can successfully navigate "noisy" environments where up to 98% of detected correspondences are false.

What Problem Does This Paper Address?

In remote sensing, "co-registration" is the process of aligning two images of the same area taken at different times or by different sensors (e.g., a satellite photo vs. a radar scan). This is notoriously difficult because different sensors "see" the world differently; a forest might look bright in an optical image but dark or textured in radar.

The traditional challenge is twofold:

  1. Reliability: Most methods struggle with "multimodal" pairs (like optical-to-radar) where intensities are inverted or obscured by noise.
  2. Validation: Usually, you only know how accurate a registration is if you have a "ground truth" map to compare it against. In real-world scenarios (like disaster management), that ground truth often doesn't exist.

The authors sought a way to register these complex pairs automatically while providing a "confidence score" (Standard Deviation of error) for the results based solely on the properties of the images themselves.

Methods

The authors introduce the RAE (Registration with Accuracy Estimation) method. Its workflow breaks down into three innovative stages:

  • Local Accuracy via CRLB: Instead of just finding matching points, the algorithm uses the Cramer–Rao Lower Bound (CRLB) to estimate the potential accuracy of every local fragment. It looks at the texture and noise of a small patch and says, "Mathematically, the best we could ever do here is X pixels of error." This CRLB value is used as a priori knowledge to filter out bad matches (outliers) before the heavy lifting begins.
  • Robust Global Estimation: The method uses an Expectation-Maximization (EM) algorithm to solve for the transformation parameters (affine or polynomial). Crucially, they integrated a Leave-One-Out Cross-Validation (LOOCV) approach. This prevents "high-leverage" points—isolated matches that might be wrong—from skewing the entire map's alignment.
  • Complexity Management: Because calculating CRLB for every possible match is computationally expensive, the authors used an incremental scheme. As the algorithm finds good matches, it "shrinks" the search zone for the next ones, keeping the overall process efficient (linearithmic complexity).

Datasets

The researchers tested RAE on eight diverse and difficult "Test Cases" (TC1–TC8) involving:

  • Sensors: Hyperion (hyperspectral), Landsat 8 (optical), ASTER GDEM-2 (Elevation models), and SIR-C (Radar).
  • Modality Mixes: Optical-to-Optical, Optical-to-DEM, Optical-to-Radar, and the rarely successful DEM-to-Radar.
  • Temporal Gaps: Images taken between 7 and 22 years apart, featuring significant changes in land cover (urban, forest, snow, and agricultural).

Key Insights and Ideas

  • Accuracy as an Input: Most researchers treat accuracy as a final grade. This paper treats it as a tool. By knowing the theoretical limit of a match's accuracy beforehand, the algorithm can prioritize "high-quality" textures and ignore "mushy" areas where noise dominates.
  • The 2% Success Rate: One of the most striking insights is the algorithm's robustness. In the optical-to-DEM case, the probability of finding a true match was as low as 2% (meaning 98% of initial guesses were wrong). Because of the CRLB-weighting and EM-optimization, the RAE method still converged on a subpixel-accurate solution.
  • No Ground Truth Needed: By deriving the covariance matrix of the transformation parameters directly from the image noise and texture, the RAE method provides a pixel-wise Standard Deviation map. This tells the user exactly which parts of the registered image are reliable.

Key Results

  • Subpixel Precision: The RAE method reached subpixel accuracy in all eight complex cases.
  • Superiority over State-of-the-Art: RAE was compared against popular methods like ImprovedSIFT, Locally Linear Transforming (LLT), and Mutual Information (MI). While MI and SIFT-based methods performed well on simple optical-to-optical pairs, they frequently failed or produced massive errors on DEM and Radar combinations where RAE remained stable.
  • Quantitative Accuracy: For optical-to-optical, RMSE was as low as 0.6 pixels. Even in the most difficult DEM-to-Radar scenarios, the error remained between 0.67 and 0.85 pixels.

Limitations and Future Work

  • Computational Speed: The authors acknowledge that RAE is roughly an order of magnitude slower than simpler feature-based methods (like SIFT) because of the intensive CRLB and EM calculations.
  • Future Directions: To address the speed issue, the authors suggest exploring multiscale (pyramid) approaches. They also propose testing more advanced similarity measures beyond Normalized Cross-Correlation (NCC) within the RAE framework to further improve results in extreme multimodal cases.

Summary updated

Abstract

This paper focuses on the potential accuracy of remote sensing (RS) image registration. We investigate how this accuracy can be estimated without ground truth available and used to improve registration quality of mono- and multimodal pair of images. At the local scale of image fragments, the Cramér-Rao lower bound (CRLB) on registration error is estimated for each local correspondence between coarsely registered pair of images. This CRLB is defined by local image texture and noise properties. Opposite to the standard approach, where registration accuracy is only evaluated at the output of the registration process, such valuable information is used by us as an additional input knowledge. It greatly helps in detecting and discarding outliers and refining the estimation of geometrical transformation model parameters. Based on these ideas, a new area-based registration method called registration with accuracy estimation (RAE) is proposed. In addition to its ability to automatically register very complex multimodal image pairs with high accuracy, the RAE method is able to provide registration accuracy at the global scale as a covariance matrix of estimation error of geometrical transformation model parameters or as pointwise registration standard deviation. This accuracy does not depend on any ground truth availability and characterizes each pair of registered images individually. Thus, the RAE method can identify image areas for which a predefined registration accuracy is guaranteed. This is essential for RS applications imposing strict constraints on registration accuracy such as change detection, image fusion, and disaster management. The RAE method is proved successful with reaching subpixel accuracy while registering eight complex mono-/multimodal and multitemporal image pairs including optical-to-optical, optical-to-radar, optical-to-digital elevation model (DEM) images, and DEM-to-radar cases. Other methods employed in comparisons fail to provide in a stable manner accurate results on the same test cases.
BibTeX
@article{uss2016multimodal,
  author = {Mikhail L. Uss and Benoît Vozel and Vladimir V. Lukin and Kacem Chehdi},
  title = {{Multimodal Remote Sensing Image Registration With Accuracy Estimation at Local and Global Scales}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2016},
  url = {https://openreview.net/forum?id=6syJpUx6Ag},
}

Loading related content...

Figure