Image Fusion Based on Multi-Directional Multiscale Analysis and Immune Optimization

Fang Liu, Jing Bai, Shuang Wang, Biao Hou, Licheng JiaoInTech 2011 · DOI · OpenAlex

Introduction

Image fusion is a critical process in computer vision and remote sensing, aimed at synthesizing information from multiple sensors into a single, high-quality representation. By merging complementary data—such as the thermal signatures from infrared sensors and the spatial detail of visible light—fusion systems can produce images that are more informative than any single source. However, traditional methods often struggle with directional features and artifacts. This paper introduces a framework that utilizes the Nonsubsampled Contourlet Transform (NSCT) and Immune Clonal Selection (ICS) to overcome these limitations.

The Limitations of Traditional Transforms

Standard Wavelet Transforms (WT) have long been the industry standard for image fusion. Despite their popularity, they possess two significant drawbacks:

  1. Isotropy: Wavelets are limited in their ability to represent highly directional features, such as curved edges and contours, which are prevalent in natural scenes.
  2. Shift-Variance: Due to downsampling, standard wavelets lack shift-invariance. This often results in "ringing" or "halo" artifacts around edges when the source images are slightly misaligned or when the transform is inverted.

While the original Contourlet Transform (CT) addressed directionality, it still suffered from downsampling-related artifacts.

Methodology: A Dual-Engine Approach

The proposed framework relies on two primary components: a structural engine for decomposition and an optimization engine for fusion weight determination.

1. Nonsubsampled Contourlet Transform (NSCT)

The NSCT serves as the structural backbone of the system. By employing nonsubsampled filter banks, the NSCT achieves full shift-invariance. This ensures that the geometric regularity of the image—its curves and contours—is preserved without the introduction of artifacts.

Nonsubsampled Contourlet Transform (NSCT) decomposition and frequency partitioning.

As shown in the figure above, the NSCT decomposes the image into a low-pass subband and multiple directional bandpass subbands at various scales. This allows the system to analyze the image's content with high directional sensitivity, capturing features that isotropic wavelets would miss.

2. Immune Clonal Selection (ICS) Optimization

Determining the optimal way to combine subbands is a complex task. Rather than using static averaging rules, the authors implement an Immune Clonal Selection (ICS) algorithm. Inspired by the biological immune system's ability to identify and clone effective antibodies, the ICS algorithm treats candidate fusion solutions as "antibodies."

The Immune Clonal Selection (ICS) algorithm flow for optimizing fusion weights.

The algorithm evolves these solutions through cycles of cloning, mutation, and selection. By maximizing a fitness function—specifically the Edge-dependent Fusion Quality Index—the ICS engine automatically determines the best weights for fusing image subbands based on the specific content of the source images.

Applications and Results

The framework was tested across two major domains: multi-sensor fusion (Visible/Infrared) and remote sensing (Panchromatic/Multispectral).

Multi-Sensor Fusion

In visible and infrared fusion tasks, the goal is to integrate heat signatures into a clear scene. The NSCT-ICS method demonstrated superior performance, achieving a Mutual Information (MI) score of 2.80, compared to 2.27 for standard Wavelet-ICS. This indicates a significantly higher retention of relevant information from the source sensors.

Remote Sensing and LHS Transform

For remote sensing, the authors combined high-resolution Panchromatic (PAN) images with lower-resolution Multispectral (MS) images. To prevent the color distortion common in standard Intensity-Hue-Saturation (IHS) transforms, they utilized a Linear LHS transform. The results were stark: the NSCT-LHS method achieved a Correlation Coefficient (CC) of 0.93, vastly outperforming PCA (0.52) and IHS (0.66).

Conclusion

By combining the multi-directional analysis of the NSCT with the adaptive optimization of the ICS algorithm, the authors have developed a robust image fusion framework. The method effectively eliminates ringing artifacts while maximizing the preservation of spectral and spatial information. Future work will focus on reducing the computational cost of the iterative ICS process and developing self-adaptive parameter tuning to further automate the fusion pipeline.

Summary updated

BibTeX
@inproceedings{liu2011image,
  author = {Fang Liu and Jing Bai and Shuang Wang and Biao Hou and Licheng Jiao},
  title = {{Image Fusion Based on Multi-Directional Multiscale Analysis and Immune Optimization}},
  booktitle = {InTech},
  year = {2011},
  doi = {10.5772/15375},
}
Figure