Multimodal Remote Sensing Image Registration with Accuracy Estimation at Local and Global Scales

Mikhail L. Uss, Benoît Vozel, Vladimir V. Lukin, Kacem Chehdi

Abstract

Accurate image registration is a fundamental prerequisite for remote sensing applications, yet estimating registration quality without ground truth remains a significant challenge. This paper introduces the Registration with Accuracy Estimation (RAE) method, an area-based approach designed for complex mono- and multi-modal image pairs. By utilizing the Cramer-Rao lower bound (CRLB) to estimate local correspondence accuracy based on image texture and noise, the RAE method treats potential accuracy as an input rather than a mere output. This allows for robust outlier detection and refined geometric transformation. Experimental results across eight diverse test cases—including optical, radar, and Digital Elevation Model (DEM) data—demonstrate that RAE consistently achieves subpixel accuracy and provides a reliable, point-wise standard deviation of error without requiring ground truth data.

Introduction

In remote sensing, co-registration—the process of aligning images of the same geographic area taken at different times or by different sensors—is critical for change detection and data fusion. However, multimodal registration (e.g., aligning an optical satellite image with a radar scan) is notoriously difficult. Different sensors capture distinct physical properties; a forest may appear bright in optical spectra but exhibit complex textures in radar, and intensities may be inverted or obscured by significant noise.

Traditional registration methods often lack a mechanism to validate accuracy in the absence of ground truth maps. Furthermore, they frequently fail when the ratio of correct matches to false correspondences is extremely low. The RAE method addresses these gaps by integrating accuracy estimation directly into the registration workflow.

Methodology: The RAE Framework

The RAE method operates on the principle that registration accuracy is dictated by local image properties. It functions across two scales:

1. Local Scale: CRLB-Based Weighting

Instead of treating all potential matches equally, RAE calculates the Cramer–Rao Lower Bound (CRLB) for every local image fragment. The CRLB defines the theoretical minimum error achievable for a given patch based on its texture and noise levels. By using this as a priori knowledge, the algorithm can prioritize high-quality textures and effectively filter out "mushy" or noisy areas where reliable matching is mathematically improbable. This allows the system to function even when up to 98% of initial correspondences are outliers.

2. Global Scale: Robust Estimation and Validation

To solve for transformation parameters (such as affine or polynomial models), RAE employs an Expectation-Maximization (EM) algorithm. To ensure the model is not skewed by "high-leverage" points—isolated matches that may be erroneous—the method incorporates Leave-One-Out Cross-Validation (LOOCV).

Crucially, the RAE method outputs a covariance matrix of the estimation error. This provides a global scale accuracy metric, often expressed as a point-wise Standard Deviation (SD) map, which informs the user exactly which regions of the registered image meet specific accuracy requirements.

Experimental Setup and Datasets

The RAE method was validated using eight complex test cases (TC1–TC8) featuring a wide array of sensors and modalities:

  • Sensors: Hyperion (hyperspectral), Landsat 8 (optical), ASTER GDEM-2 (elevation models), and SIR-C (radar).
  • Modalities: Optical-to-Optical, Optical-to-DEM, Optical-to-Radar, and DEM-to-Radar.
  • Temporal Variance: The dataset included images taken up to 22 years apart, encompassing significant land-cover changes in urban, forested, and agricultural environments.

Key Results and Discussion

The RAE method demonstrated superior stability and precision compared to state-of-the-art techniques like ImprovedSIFT, Locally Linear Transforming (LLT), and Mutual Information (MI). While traditional methods often failed on difficult DEM-to-Radar or Optical-to-Radar pairs, RAE remained stable.

  • Subpixel Accuracy: RAE achieved subpixel precision across all eight cases. For optical-to-optical pairs, the Root Mean Square Error (RMSE) was as low as 0.6 pixels. In the most challenging DEM-to-Radar scenarios, error remained between 0.67 and 0.85 pixels.
  • Robustness to Outliers: In the optical-to-DEM test case, the probability of a true match was only 2%. Despite this, the CRLB-weighted EM optimization successfully converged on an accurate solution.
  • Self-Validation: The ability to generate a point-wise registration SD map without ground truth is a significant advancement for real-world applications where reference data is unavailable.

Conclusion

The RAE method represents a shift in remote sensing registration by treating accuracy as a functional input. By leveraging the mathematical limits of local image patches via CRLB, it achieves high-precision alignment across vastly different sensor modalities. While the computational cost is higher than simpler feature-based methods due to the intensive EM and CRLB calculations, the resulting reliability and built-in confidence metrics make it a powerful tool for complex satellite data analysis. Future work will focus on optimizing computational speed through multiscale pyramid approaches and exploring advanced similarity measures.

Summary updated

Abstract

This paper focuses on potential accuracy of remote sensing images registration. We investigate how this accuracy can be estimated without ground truth available and used to improve registration quality of mono- and multi-modal pair of images. At the local scale of image fragments, the Cramer-Rao lower bound (CRLB) on registration error is estimated for each local correspondence between coarsely registered pair of images. This CRLB is defined by local image texture and noise properties. Opposite to the standard approach, where registration accuracy is only evaluated at the output of the registration process, such valuable information is used by us as an additional input knowledge. It greatly helps detecting and discarding outliers and refining the estimation of geometrical transformation model parameters. Based on these ideas, a new area-based registration method called RAE (Registration with Accuracy Estimation) is proposed. In addition to its ability to automatically register very complex multimodal image pairs with high accuracy, the RAE method provides registration accuracy at the global scale as covariance matrix of estimation error of geometrical transformation model parameters or as point-wise registration Standard Deviation. This accuracy does not depend on any ground truth availability and characterizes each pair of registered images individually. Thus, the RAE method can identify image areas for which a predefined registration accuracy is guaranteed. The RAE method is proved successful with reaching subpixel accuracy while registering eight complex mono/multimodal and multitemporal image pairs including optical to optical, optical to radar, optical to Digital Elevation Model (DEM) images and DEM to radar cases. Other methods employed in comparisons fail to provide in a stable manner accurate results on the same test cases.
BibTeX
@article{uss2016multimodal,
  author = {Mikhail L. Uss and Benoît Vozel and Vladimir V. Lukin and Kacem Chehdi},
  title = {{Multimodal Remote Sensing Image Registration with Accuracy Estimation at Local and Global Scales}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2016},
  url = {https://openreview.net/forum?id=6syJpUx6Ag},
}
Figure