TL;DR
This paper introduces DePF, a novel infrared and visible image fusion method that addresses the common issues of texture loss and edge information deficiency. DePF utilizes a new "decomposition pooling" (de-pooling) mechanism to extract multi-scale and detail features simultaneously, preserving rich information during fusion and reconstruction.
What Problem Does This Paper Address?
Infrared and visible image fusion aims to create a single image that combines the thermal radiation information from infrared images with the rich texture and detail from visible images. Existing fusion methods often struggle with losing fine texture details or edge information, leading to suboptimal results. Traditional pooling operations like max-pooling, while useful for downsampling, can discard crucial information. Furthermore, many deep learning fusion methods either rely on complex, manually designed fusion strategies or suffer from unstable training and information loss during reconstruction due to the lack of exact inverses for pooling operations. This paper tackles these challenges by proposing a new pooling mechanism that preserves more information and a more robust fusion framework.
Methods
The core innovation of DePF is the decomposition pooling (de-pooling) mechanism. Unlike traditional pooling, de-pooling uses four specialized kernels (MS, VD, HD, DD) to extract both multi-scale structural information (MS) and detailed texture information (VD, HD, DD for vertical, horizontal, and diagonal directions, respectively). This is inspired by the Prewitt operator. The de-pooling operation is designed to have an exact inverse, de-unpooling, which allows for precise reconstruction of features without information loss or the need for post-processing.
The DePF architecture consists of:
- Encoder: Employs convolutional blocks and de-pooling to extract multi-scale and detail features from both infrared and visible images.
- Fusion Layer: Utilizes a spatial attention model to adaptively fuse the multi-scale deep features extracted by the encoder. Detail features from corresponding scales are combined using an addition strategy.
- Decoder: Uses convolutional blocks and de-unpooling to reconstruct the fused image, leveraging the preserved multi-scale and detail information.
The training strategy involves pre-training the encoder and decoder as an auto-encoder without the fusion strategy, using pixel and structural similarity (SSIM) losses to ensure accurate reconstruction.
Datasets
The paper uses the following datasets for training and evaluation:
- MS-COCO: Used for pre-training the auto-encoder network.
- MSRS Dataset: Used for qualitative and quantitative evaluation.
- LLVIP Dataset: Used for qualitative and quantitative evaluation.
- TNO Dataset: Used for qualitative and quantitative evaluation.
Key Insights and Ideas
- De-pooling for Information Preservation: The de-pooling mechanism is a significant advancement over traditional pooling methods. By extracting multiple feature types (multi-scale and directional details) simultaneously and having an exact inverse, it drastically reduces information loss during feature extraction and reconstruction.
- De-unpooling for Perfect Reconstruction: The existence of an exact inverse for de-pooling (de-unpooling) is crucial. It allows the decoder to perfectly reconstruct features, leading to sharper and more artifact-free fused images, a common problem with methods relying on max-pooling.
- Spatial Attention for Adaptive Fusion: Employing a spatial attention model for fusion allows the network to adaptively weigh features from different modalities, ensuring that important information from both infrared and visible images is effectively combined.
- Kernel Design Matters: The paper highlights the importance of kernel design, showing that using 4x4 kernels in de-pooling (compared to 2x2) leads to better performance by capturing a larger receptive field and richer information.
Key Results
The DePF method consistently outperforms nine state-of-the-art fusion methods across multiple datasets (MSRS, LLVIP, TNO) in both qualitative and quantitative evaluations.
- Quantitative: DePF ranks first or second in most metrics (SD, VIF, AG, SCD, EN) across the datasets, demonstrating superior contrast, information fidelity, detail retention, and correlation with source images. For example, on the MSRS dataset, DePF achieved the best scores in SD (8.4779), AG (4.0498), SCD (1.7159), and EN (6.7697).
- Qualitative: Visual comparisons show that DePF successfully preserves texture details, maintains structure intensity, and avoids artifacts and noise, which are prevalent in the outputs of other methods, especially in challenging regions highlighted in the figures.
Limitations and Future Work
- Dataset Specificity: While DePF performs well on the tested datasets, the paper notes that on the TNO dataset, where visible images have less obvious texture details, the method's preference for preserving texture might not yield as dramatic improvements compared to other datasets. This suggests that the performance might be influenced by the inherent detail richness of the source images.
- Computational Cost: While not explicitly detailed as a limitation, the use of 4x4 kernels and multiple convolutional blocks implies a certain computational cost, though the paper emphasizes the efficiency of the fusion strategy.
- Future Work: The paper suggests that the proposed de-pooling mechanism and fusion framework can be further explored and adapted for other image fusion tasks beyond infrared and visible images. The effectiveness of the spatial attention fusion strategy also opens avenues for further research in adaptive fusion techniques.
Summary updated