CHITNet: A Complementary to Harmonious Information Transfer Network for Infrared and Visible Image Fusion

Yafei Zhang, Keying Du, Huafeng Li, Zhengtao Yu, Yu LiuIEEE Trans. Instrumentation & Measurement 2023 · OpenReview

TL;DR

This paper introduces CHITNet, a novel network for fusing infrared and visible images. Instead of focusing on the difficult task of extracting complementary features, CHITNet shifts the paradigm to transferring complementary information into a "harmonious" representation, integrating shared and modality-specific features for higher quality fused images.

What Problem Does This Paper Address?

Infrared and visible image fusion (IVIF) aims to combine the strengths of both modalities: visible images offer high spatial resolution and rich texture, while infrared images excel in low-light and adverse weather conditions by capturing thermal information. The core challenge in IVIF is effectively extracting and merging the complementary information present in these two distinct image types. Existing methods often struggle with this, leading to complex fusion strategies that can result in lost details or artifacts. This paper addresses the difficulty of complementary feature extraction by proposing a new approach that focuses on creating a "harmonious" representation.

Methods

CHITNet is built upon three key modules:

  • Mutual Information Transfer (MIT) Module: This module is the heart of the "complementary to harmonious" transfer. It involves:
  • Image Inversion: Inverting images to make salient targets in one modality resemble those in the other, facilitating channel concatenation.
  • Feature Extraction (FE): Using a residual dense network (RDN) to extract initial features.
  • Mutual Representation Learning (MRL): Employing an attention-based mechanism (similar to Transformers) to allow features from one modality to query and represent features from the other, effectively transferring complementary information into a harmonious representation.
  • Structure Information Preserved Harmonious Information Acquisition (SIPHIA) Module: This module ensures that the transferred harmonious features are rich and that crucial structural information (like edges) from the source images is preserved. It consists of:
  • Harmonious Information Acquisition Supervised by Source Image (HIASSI): This sub-module reconstructs parts of the transferred features and uses a reconstruction loss against the original source images to ensure the harmonious features retain valuable information.
  • Structure Information Preservation (SIP): This sub-module uses cross-attention and gradient loss to explicitly preserve edge details from the source images.
  • Fusion Result Reconstruction (FRR) Module: A decoder that takes the combined features from MIT and SIPHIA to generate the final fused image.

A Mutual Promotion Training Paradigm (MPTP) is also introduced, where MIT and SIPHIA modules are trained iteratively to improve their collaboration and overall performance.

Datasets

  • Training: RoadScene dataset (148 image pairs), augmented by random cropping to 120x120 patches, resulting in 148,000 training pairs.
  • Testing:
    • RoadScene dataset (38 pairs)
    • TNO dataset (40 pairs)
    • VOT2020-RGBT dataset (40 pairs)
    • OTCBVS dataset (40 pairs)
    • LLVIP dataset (for generalization tests)
    • CT-MRI dataset (for generalization tests)

Key Insights and Ideas

  • Shifting the Paradigm: The core insight is that directly extracting complementary features is difficult. By focusing on transferring complementary information into a "harmonious" representation, the problem becomes more tractable and leads to better results.
  • Harmonious Features: The concept of "harmonious features" as a comprehensive representation that includes both shared and modality-specific information is key. This avoids losing information present in either source image.
  • Mutual Promotion Training: Training the MIT and SIPHIA modules iteratively (MPTP) significantly improves performance, highlighting the importance of their synergistic interaction.
  • Edge Preservation is Crucial: The SIP module's explicit focus on preserving edge structure information is vital for high-quality fusion, as demonstrated by ablation studies.

Key Results

  • Quantitative Performance: CHITNet achieves state-of-the-art (SOTA) performance across multiple metrics (CC, EN, QAB/F, SCD, SSIM) on various datasets, often ranking first or second. For instance, on the RoadScene dataset, it ranks first in three metrics and second in another.
  • Qualitative Performance: Visual comparisons show that CHITNet produces fused images with clearer edges, richer textures, and better contrast compared to SOTA methods. It effectively preserves details from both infrared and visible images, avoiding common issues like lost cloud textures or blurred edges.
  • Efficiency: CHITNet demonstrates good computational efficiency with low FLOPs and a small parameter size, making it suitable for practical applications.

Limitations and Future Work

  • Robustness to Noise: The paper acknowledges that the model's performance can be negatively impacted by noise in the input images. Improving robustness to various forms of degradation is identified as a limitation and a direction for future research.
  • Generalization: While the paper shows good generalization to other fusion tasks (LLVIP, CT-MRI), further work could explore more diverse datasets and fusion scenarios.
  • Hyperparameter Sensitivity: The paper analyzes the sensitivity to hyperparameters like λ_edge and λ_JG, suggesting that optimal values exist for balancing edge and texture preservation. Further research could explore automated hyperparameter tuning.

Summary updated

Abstract

Current infrared and visible image fusion (IVIF) methods go to great lengths to excavate complementary features and design complex fusion strategies, which is extremely challenging. To this end, we rethink the IVIF outside the box, proposing a complementary to harmonious information transfer network (CHITNet). It reasonably transfers complementary information into harmonious one, which integrates both the shared and complementary features from two modalities. Specifically, to skillfully sidestep aggregating complementary information in IVIF, we design a mutual information transfer (MIT) module to mutually represent features from two modalities, roughly transferring complementary information into harmonious one. Then, a harmonious information acquisition supervised by source image (HIASSI) module is devised to further ensure the complementary to harmonious information transfer after MIT. Meanwhile, we also propose a structure information preservation (SIP) module to guarantee that the edge structure information of the source images can be transferred to the fusion results. Moreover, a mutual promotion training paradigm with interaction loss is adopted to facilitate better collaboration among MIT, HIASSI and SIP. In this way, the proposed method is able to generate fused images with higher qualities. Extensive experimental results demonstrate the superiority of CHITNet over state-of-the-art algorithms in terms of visual quality and quantitative evaluations.
BibTeX
@article{zhang2023chitnet,
  author = {Yafei Zhang and Keying Du and Huafeng Li and Zhengtao Yu and Yu Liu},
  title = {{CHITNet: A Complementary to Harmonious Information Transfer Network for Infrared and Visible Image Fusion}},
  journal = {IEEE Trans. Instrumentation \& Measurement},
  year = {2023},
  url = {https://openreview.net/forum?id=4Ejb33vBC1},
}
Figure