Infrared and Visible Image Fusion via Interactive Compensatory Attention Adversarial Learning

Zhishe Wang, Wenyu Shao, Yanlin Chen, Jiawei Xu, Xiaoqin Zhang

TL;DR

This paper introduces ICAFusion, a novel generative adversarial learning framework for fusing infrared and visible images. It addresses the limitations of existing methods by incorporating interactive and compensatory attention modules to better capture both global and local features, leading to more balanced and visually superior fused images.

What Problem Does This Paper Address?

Infrared (IR) and visible (VIS) images offer complementary information: IR excels at detecting targets in low-light or adverse conditions by capturing heat signatures, while VIS provides rich texture and detail through light reflection. Effectively fusing these modalities is crucial for applications like RGBT tracking and object detection. However, current fusion methods often struggle with this balance. Many methods simply concatenate source images or their features and rely on standard convolutional operations, neglecting global context. This can lead to fused images that are biased towards one modality, either losing important IR target information or blurring VIS textures. Existing adversarial methods, while improving balance, can still suffer from insufficient local detail and blurred edges.

Methods

The core of ICAFusion is a generative adversarial network (GAN) architecture designed for end-to-end image fusion.

  • Generator Architecture: The generator features a multi-level encoder-decoder structure with a triple path: one for IR, one for VIS, and a third for their concatenation. The IR and VIS paths provide complementary intensity and gradient information to the concatenated path, preserving crucial details from both sources.
  • Interactive and Compensatory Attention Modules: This is the key innovation. These modules, inspired by CBAM, are designed to model long-range dependencies and facilitate feature communication between the triple paths.
  • Interactive Attention: Applied in the encoder, it allows the IR and VIS paths to exchange information, enhancing feature representation.
  • Compensatory Attention: Used in the decoder, it compensates for lost information during feature reconstruction, ensuring that both IR and VIS characteristics are well-represented in the final fused image. These modules combine channel and spatial attention mechanisms.
  • Dual Discriminators: To ensure a balanced fusion, ICAFusion employs two discriminators: Discriminator-IR and Discriminator-VIS. These discriminators are trained to distinguish between the fused image and the original IR and VIS images, respectively. This forces the generator to produce a fused image that closely resembles both source modalities in terms of pixel distribution and texture details.
  • Loss Function: The loss function includes an adversarial loss (to fool the discriminators) and a content loss that uses Frobenius norm and L1 norm to preserve pixel intensity and gradient variations from both source images.

Datasets

The paper uses the following datasets for training and evaluation:

  • Training: TNO dataset (25 IR/VIS pairs).
  • Evaluation: TNO, Roadscene, and OTCBVS datasets.

Key Insights and Ideas

  • Bridging Global and Local Features: The paper highlights that simply extracting local features via convolutions is insufficient. By incorporating attention mechanisms that model long-range dependencies, ICAFusion can better integrate global context with local details.
  • Complementary Attention: The distinction between "interactive" (encoder) and "compensatory" (decoder) attention is crucial. This suggests a staged approach to feature fusion, where interaction enhances initial feature extraction, and compensation refines the reconstruction process.
  • Dual Discriminators for Balanced Fusion: Using separate discriminators for IR and VIS ensures that the generator is pushed to satisfy the distinct characteristics of both modalities, leading to a more balanced output than single-discriminator GANs.
  • End-to-End Adversarial Learning: The adoption of an end-to-end adversarial framework eliminates the need for manual fusion strategies, making the system more adaptable and potentially more robust.

Key Results

  • Subjective Evaluation: Visual comparisons across TNO, Roadscene, and OTCBVS datasets consistently show ICAFusion producing fused images with higher brightness for IR targets, richer VIS textures, and better overall contrast and visual perception compared to state-of-the-art methods like MDLatLRR, DenseFuse, FusionGAN, and others.
  • Objective Evaluation: ICAFusion achieved top ranks in several objective metrics (e.g., AG, EN, SD, MI, SF, NCIE, Qabf, VIF) across the tested datasets, demonstrating superior fusion performance. For instance, on the TNO dataset, it ranked first in AG, EN, MI, SF, NCIE, and Qabf.
  • Computational Efficiency: While not the fastest, ICAFusion demonstrates competitive fusion efficiency, slightly lower than DenseFuse and IFCNN, which use simpler fusion rules.

Limitations and Future Work

  • Computational Efficiency: While competitive, ICAFusion is not the absolute fastest method. Future work could focus on optimizing the network architecture for even greater speed.
  • Future Directions: The authors plan to further optimize the network architecture and explore incorporating attention mechanisms into the discriminator. They also aim to extend the framework to other fusion tasks, such as multi-band, multi-exposure, and multi-focus image fusion.

Summary updated

Abstract

The existing generative adversarial fusion methods generally concatenate source images or deep features, and extract local features through convolutional operations without considering their global characteristics, which tends to produce a limited fusion performance. Toward this end, we propose a novel interactive compensatory attention fusion network, termed ICAFusion. In particular, in the generator, we construct a multi-level encoder-decoder network with a triple path, and design infrared and visible paths to provide additional intensity and gradient information for the concatenating path. Moreover, we develop the interactive and compensatory attention modules to communicate their pathwise information, and model their long-range dependencies through a cascading channel-spatial model. The generated attention maps can more focus on infrared target perception and visible detail characterization, and are used to reconstruct the fusion image. Therefore, the generator takes full advantage of local and global features to further increase the representation ability of feature extraction and feature reconstruction. Extensive experiments illustrate that our ICAFusion obtains superior fusion performance and better generalization ability, which precedes other advanced methods in the subjective visual description and objective metric evaluation. Our codes will be public at <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><uri>https://github.com/Zhishe-Wang/ICAFusion</uri></i> .
BibTeX
@article{wang2022infrared,
  author = {Zhishe Wang and Wenyu Shao and Yanlin Chen and Jiawei Xu and Xiaoqin Zhang},
  title = {{Infrared and Visible Image Fusion via Interactive Compensatory Attention Adversarial Learning}},
  journal = {IEEE Transactions on Multimedia},
  year = {2022},
}
Figure