An advanced image fusion algorithm based on wavelet transform: incorporation with PCA and morphological processing

Yufeng Zheng, Edward A. Essock, Bruce C. HansenImage Processing: Algorithms and Systems 2004 · DOI

TL;DR

This paper introduces an advanced image fusion algorithm (aDWT) that combines discrete wavelet transform (DWT) with principal component analysis (PCA) and morphological processing. The proposed aDWT method demonstrates superior performance across various image types compared to existing fusion techniques, offering a more robust solution for combining heterogeneous imagery.

What Problem Does This Paper Address?

Image fusion, the process of combining multiple images into a single, more informative image, is crucial for applications like medical imaging, remote sensing, and nighttime operations. While various fusion techniques exist, a significant challenge lies in effectively merging images with differing characteristics such as intensity, contrast, and noise levels. Existing methods, often based on multiresolution analysis like DWT or pyramid transforms, can struggle to preserve all relevant information while avoiding artifacts. This paper aims to address this by developing a more advanced fusion algorithm that can handle such inhomogeneous imagery more effectively.

Methods

The core of this paper is the advanced DWT (aDWT) method. It builds upon the standard DWT fusion approach, which typically involves fusing approximation coefficients (e.g., by averaging) and detail coefficients (e.g., by selecting the largest absolute values) at different scales. The aDWT introduces two key enhancements:

  • Principal Component Analysis (PCA) for Approximation Coefficients: At the highest transform scale, PCA is applied to the approximation coefficients of the two input images. The principal eigenvector is then used to fuse these coefficients. This helps to capture the dominant variance and relationships between the input images.
  • Morphological Processing for Detail Coefficients: For the detail coefficients (horizontal, vertical, and diagonal), after selecting the coefficients with larger absolute values, a neighborhood morphological processing procedure (specifically "filling" and "cleaning" operations on a 3x3 region) is applied. This step aims to verify the selected pixels, reduce inconsistencies, and minimize distortion in the fused image.

The paper also reviews standard DWT and PCA techniques, as well as several pyramid-based fusion methods (Laplacian, Ratio of low-pass, Gradient, and Morphological pyramids) for comparison.

Datasets

The authors evaluated their proposed aDWT method using four types of imagery:

  1. Two simulated off-focus image pairs: These were generated by blurring a reference "Lena" image with different Gaussian windows and slight gray-level reductions, creating "Lena-I1" and "Lena-I2." These datasets had ground truth available.
  2. Frequently used samples: This category included:
  3. Two clocks captured with different focal planes.
  4. Two types of medical imagery (CT and MRI).
  5. A pair of remote sensing images (infrared and low-light visible).
  6. Image intensified and infrared paired images: A dataset of 28 pairs of realistic night-vision images from the Army NVESD. These images were moderately noisy and underwent a 2D median filter as preprocessing.
  7. Near infrared and infrared paired images: A dataset of 50 pairs captured indoors and outdoors using their own facilities. These also underwent 2D median filtering for noise reduction.

Key Insights and Ideas

  • PCA enhances approximation coefficient fusion: Using PCA for approximation coefficients allows for a more sophisticated fusion that considers the principal components of the input data, potentially leading to better preservation of structural information.
  • Morphological processing improves detail fusion: Applying morphological operations to detail coefficients helps to refine the selection of important features, reduce noise-induced artifacts, and improve the spatial consistency of the fused image.
  • IQI is a reliable measure for images without ground truth: The Image Quality Index (IQI) proved to be a trustworthy metric for evaluating fusion performance when a ground truth image is unavailable, as it correlates well with RMSE.
  • aDWT excels with inhomogeneous imagery: The proposed aDWT method consistently outperformed other algorithms, especially when dealing with images that have significant differences in intensity, contrast, and noise.

Key Results

The aDWT method demonstrated superior performance across various image types and evaluation metrics.

  • Simulated Images (Lena-I1, Lena-I2): For Lena-I1, aDWT achieved the best results for Spatial Frequency (SF) and Image Quality Index (IQI), while the contrast pyramid was best for RMSE. For Lena-I2, aDWT performed best on three measures (RMSE, SF, IQI), with the Laplacian pyramid leading in entropy.
  • Frequently Used Samples: The aDWT was generally the best performer across most metrics, particularly for medical and remote sensing imagery. It was slightly outperformed by Laplacian's IQI on remote sensing data but was still significantly better than regular DWT.
  • Army NVESD Imagery: While Laplacian pyramid had the best IQI, aDWT outperformed other methods on the other two metrics (entropy and SF).
  • Lab Imagery (Near IR/IR): The aDWT achieved the best results across all three measures (entropy, SF, IQI) for this dataset.
  • Qualitative Assessment: Perceptually, the fused images produced by the aDWT method generally appeared to be the best.

Limitations and Future Work

  • Wavelet Order and Decomposition Level: The authors chose the fourth order of Symlets and a specific decomposition level ($l_0=4$) to balance sharpness and distortion. They note that different choices might yield different results, and further exploration could be beneficial.
  • Entropy as a Measure: The paper highlights that entropy is not always a reliable measure for image fusion quality, especially when the reference image's entropy is smaller than the input images'.
  • Spatial Frequency (SF) Limitations: While SF is considered more reliable than entropy, the authors suggest that it, along with RMSE, may not fully distinguish useful information from distortion. They propose developing a new enhanced measure based on the current SF definition in a subsequent paper.
  • Future Work: The paper implicitly suggests that further refinement of image quality metrics is needed. The aDWT's success with inhomogeneous imagery positions it as a promising method for future advancements in image fusion.

Summary updated

BibTeX
@inproceedings{zheng2004advanced,
  author = {Yufeng Zheng and Edward A. Essock and Bruce C. Hansen},
  title = {{An advanced image fusion algorithm based on wavelet transform: incorporation with PCA and morphological processing}},
  booktitle = {Image Processing: Algorithms and Systems},
  year = {2004},
  doi = {10.1117/12.523966},
}
Figure