An improved mutual information similarity measure for registration of multi-modal remote sensing images

Maha Shadaydeh, Tamás SzirányiProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2015 · DOI

TL;DR

This paper introduces a novel Weighted Joint Histogram (WJH) approach to improve Mutual Information (MI) based image registration for multi-modal remote sensing images. By assigning weights to pixels based on their gradient and proximity to key points, the method effectively reduces spurious local maxima in the similarity measure, leading to more accurate and robust registration.

What Problem Does This Paper Address?

Registering multi-modal remote sensing images, meaning images of the same area taken at different times or with different sensors (e.g., optical vs. radar), is crucial for applications like change detection and image fusion. Mutual Information (MI) is a popular metric for this task because it measures the statistical dependence between image intensities, making it suitable for different sensor types. However, traditional MI has two major drawbacks: its similarity surface often has many "false peaks" (local maxima) that can mislead the registration process, and it completely ignores spatial information, treating pixels independently. This can lead to inaccurate alignments, especially in areas with uniform intensity.

Methods

The core innovation is the Weighted Joint Histogram (WJH) approach. Instead of treating all pixels equally when building the joint histogram (which estimates the joint probability distribution), the proposed method assigns a weight to each pixel in the reference image. This weight is calculated using an exponential function that considers two factors:

  • Normalized Gradient Image: Pixels with higher gradients (edges, texture) are given more weight, as they often contain more discriminative information for registration.
  • Distance Image: Pixels closer to user-defined "key points" (regions of interest) are also given higher weights. These key points can be manually selected or automatically detected using algorithms like Harris or SIFT.

The joint histogram is then computed by summing these weights for pixels falling into each intensity bin, rather than simply counting pixel occurrences. This effectively creates an "irregular sampling" of the image, focusing more on informative areas. The authors then apply this WJH to both Mutual Information (WJH-MI) and Normalized Mutual Information (WJH-NMI) calculations.

Datasets

The authors evaluated their method using two sets of real-world remote sensing image pairs from the Szada area in Hungary:

  1. Experiment 1: A pair of images from 1984 and 2007, with the 1984 image deliberately translated and rotated. This allowed for a ground truth transformation to evaluate accuracy.
  2. Experiment 2: A pair of images from 2000 and 2007. For this experiment, the true transformation was unknown, and a coarse-to-fine strategy was employed.

Key Insights and Ideas

  • Spatial Information is Key: The paper strongly emphasizes that incorporating spatial information (gradients and proximity to key points) is crucial for overcoming the limitations of traditional MI.
  • Focusing on Informative Pixels: By weighting pixels, the method effectively prioritizes areas with high gradients and those near key points, which are generally more reliable for alignment. This reduces the influence of noisy or homogenous regions.
  • Smoother Similarity Surfaces: The weighting scheme leads to similarity measure surfaces with more distinct peaks, making it easier for optimization algorithms to find the correct global maximum and reducing the chance of getting stuck in local optima.
  • Computational Efficiency: The gradient and distance images are calculated once for the reference image and are independent of the geometric transformation being tested. This means the proposed weighting scheme adds minimal computational overhead compared to conventional MI.

Key Results

The proposed WJH-MI and WJH-NMI methods consistently outperformed conventional MI and NMI in their experiments.

  • Reduced Registration Errors: In Experiment 1, the WJH methods achieved significantly lower registration errors (translation and rotation) compared to the standard MI/NMI methods. For example, the pixel errors for translation were reduced from 4 and 13 to 0 and 1, respectively.
  • More Distinguished Peaks: Contour plots of the similarity surfaces (Figure 2 and 5) clearly show that the WJH methods produce sharper, more pronounced peaks, indicating a more reliable convergence to the correct transformation.
  • Better Matching Accuracy: In Experiment 2, visual inspection of the registered images (Figure 4) and contour plots (Figure 5) demonstrated that WJH-NMI achieved considerably better matching accuracy, with a single dominant peak compared to multiple local maxima for NMI.

Limitations and Future Work

  • Key Point Selection: The performance of the WJH method relies on the selection of key points. While automatic methods are mentioned, the paper primarily uses arbitrary selection in experiments. Further investigation into optimal key point selection strategies could be beneficial.
  • Parameter Tuning: The constant 'k' in the weight function and the variance 'σ' for the LoG filter are parameters that need to be set. The paper mentions 'k' can be proportional to image resolution and 'σ' was set to 1 for experiments, but a more systematic analysis of parameter sensitivity might be warranted.
  • Multi-scale Analysis: The authors suggest that extending the method to multi-scale analysis using different LoG filter parameters is a promising direction for future research.

Summary updated

Abstract

Registration of multi-modal remote sensing images is an essential and challenging task in different remote sensing applications such as image fusion and multi-temporal change detection. Mutual Information (MI) has shown to be successful similarity measure for multi-modal image registration applications, however it has some drawbacks. 1. MI surface is highly non-convex with many local maxima. 2. Spatial information is completely lost in the calculation of the joint intensity probability distribution. In this paper, we present an improved MI similarity measure based on a new concept in integrating other image features as well as spatial information in the estimation of the joint intensity histogram which is used as an estimate of the joint probability distribution. The proposed method is based on the idea that each pixel in the reference image is assigned a weight, then each bin in the joint histogram is calculated as the summations of the weights of the pixels corresponding to that bin. The weight given to each pixel in the reference image is an exponential function of the corresponding pixel values in a distance image and a normalized gradient image such that higher weights are given to points close to one or more selected key points as well as points with high normalized gradient values. The proposed method is in essence a kind of calculating similarity measure using irregular sampling where sampling frequency is higher in areas close to key-points or areas with higher gradients. We have compared the proposed method with the conventional MI and Normalized MI methods for registration of pairs of multi-temporal multi-modal remote sensing images. We observed that the proposed method produces considerably better registration function containing fewer erroneous maxima and leading to higher success rate.
BibTeX
@inproceedings{shadaydeh2015improved,
  author = {Maha Shadaydeh and Tamás Szirányi},
  title = {{An improved mutual information similarity measure for registration of multi-modal remote sensing images}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2015},
  doi = {10.1117/12.2194319},
}
Figure