Combining Regional Energy and Intuitionistic Fuzzy Sets for Infrared and Visible Image Fusion

Xiaoxue Xing, Cong Luo, Jian Zhou, Minghan Yan, Cheng Liu, Tingfa Xu2021 · DOI

TL;DR

This paper introduces a novel image fusion method that combines regional energy (RE) and intuitionistic fuzzy sets (IFS) within the non-subsampled shearlet transform (NSST) framework. The goal is to create fused images that enhance both target visibility and texture details, outperforming existing methods on standard datasets.

What Problem Does This Paper Address?

Infrared (IR) and visible (VIS) images capture complementary information. IR images excel at detecting heat signatures but often lack spatial resolution and background detail. VIS images, conversely, provide rich texture and high spatial resolution but are limited in low-light or obscured conditions. Image fusion aims to combine these strengths into a single, more informative image, crucial for applications like target detection, surveillance, and medical imaging. However, achieving a balance between preserving salient IR targets and rich VIS textures while minimizing artifacts remains a challenge. Existing fusion methods often struggle to effectively transfer both types of information without degradation.

Methods

The proposed method, RE-IFS-NSST, leverages the strengths of NSST for multi-scale decomposition, regional energy for target saliency, and intuitionistic fuzzy sets for robust information transfer.

  1. NSST Decomposition: The IR and VIS images are decomposed into low-frequency (background and overall structure) and high-frequency (details, edges, textures) sub-bands using the Non-Subsampled Shearlet Transform (NSST). This transform is chosen for its ability to capture multi-scale and multi-directional information.
  2. Low-Frequency Fusion: This is a two-step process:
  3. Regional Energy (RE) Pre-fusion: An RE-based fusion rule is applied to the low-frequency sub-bands. RE is calculated by summing the energy within a local neighborhood, which helps identify regions with significant target information, particularly in IR images. This step aims to preserve the salient target information.
  4. Intuitionistic Fuzzy Sets (IFS) Final Fusion: The pre-fused low-frequency image is then fused using IFS. IFS, an extension of fuzzy sets, accounts for membership, non-membership, and hesitation degrees, making it more robust in handling uncertainty and vagueness. This step is designed to transfer texture information from VIS images more effectively.
  5. High-Frequency Fusion: The 'max-absolute' fusion rule is applied to the high-frequency sub-bands. This rule selects the coefficients with the largest absolute values, which are typically associated with strong edges and texture details, thus preserving fine features.
  6. NSST Reconstruction: Finally, the fused low-frequency and high-frequency sub-bands are reconstructed into a single fused image using the inverse NSST transform.

Datasets

The proposed method was evaluated on two publicly available datasets:

  • TNO Image Fusion Dataset: Used for testing the method's effectiveness on various IR and VIS image pairs.
  • RoadScene Dataset: Used to further evaluate the method's applicability, particularly on traffic scenes with color VIS images (which were converted to grayscale for fusion).

Key Insights and Ideas

  • Synergy of RE and IFS: The core innovation lies in combining RE and IFS. RE effectively highlights IR targets, while IFS provides a more nuanced way to transfer VIS texture information by handling uncertainty. This dual approach addresses the limitations of using either method alone.
  • NSST's Multi-Scale Advantage: The use of NSST is crucial for decomposing images into relevant frequency bands, allowing for specialized fusion rules to be applied to each. Its ability to capture directional information is also beneficial for preserving image details.
  • Target vs. Texture Trade-off: The paper demonstrates a successful strategy to balance the often conflicting goals of preserving strong IR targets and detailed VIS textures. The two-step low-frequency fusion is key to this balance.
  • Robustness to Uncertainty: IFS's ability to handle hesitation degrees makes it well-suited for image fusion, where pixel information can be ambiguous. This contributes to a more robust and less artifact-prone fusion process.

Key Results

  • Qualitative Improvements: The fused images produced by the RE-IFS-NSST method exhibit more obvious targets, higher contrast, and richer detailed information compared to other methods. They are described as being more suitable for human visual observation.
  • Quantitative Performance:
    • On the TNO dataset, the proposed method outperformed RE-NSST and IFS-NSST in most quantitative metrics (E, MI, CE, SPD, PSNR), demonstrating superior comprehensive performance.
  • When compared to nine other state-of-the-art methods on the TNO dataset, the proposed method achieved the best performance in terms of Entropy (E), Average Gradient (AG), Spectral Distortion (SPD), and Peak Signal-to-Noise Ratio (PSNR). It also achieved the second-best performance in Mutual Information (MI) and Cross Entropy (CE).
  • Similar results were observed on the RoadScene dataset, where the proposed method again led in E, AG, SPD, and PSNR, and was second-best in MI and CE.
  • Computational Complexity: While not the fastest method, the proposed RE-IFS-NSST method offers a good balance between fusion quality and computational cost, demonstrating stability across different datasets.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that the proposed method is not the fastest among those compared. While the fusion quality is superior, further optimization for speed could be beneficial.
  • Parameter Tuning: The paper mentions setting empirical values for parameters like k1 and k2 in the IFS calculation. Future work could explore adaptive methods for parameter selection to further enhance robustness.
  • Dataset Specificity: While tested on two diverse datasets, exploring performance on a wider range of IR/VIS image types (e.g., different sensor characteristics, varying illumination conditions) could be valuable.

The authors suggest that the method has good applications in target detection, military, medical diagnosis, and target tracking, implying these are areas for future application and development.

Summary updated

BibTeX
@misc{xing2021combining,
  author = {Xiaoxue Xing and Cong Luo and Jian Zhou and Minghan Yan and Cheng Liu and Tingfa Xu},
  title = {{Combining Regional Energy and Intuitionistic Fuzzy Sets for Infrared and Visible Image Fusion}},
  year = {2021},
  doi = {10.3390/s21237813},
}
Figure