TL;DR
This paper introduces DAE-Fuse, a novel two-phase autoencoder framework for multi-modality image fusion that generates sharper, more natural fused images by adaptively learning from different modalities. It addresses limitations of existing methods that produce blurry images or exhibit modality bias, and extends image fusion to video while preserving temporal consistency.
What Problem Does This Paper Address?
Reliable perception is crucial in challenging environments like nighttime or low-visibility conditions, where multi-modality image fusion, especially integrating infrared and visible light, offers a robust solution. However, current fusion methods struggle with common issues: GAN-based approaches often result in blurry images lacking fine details, while autoencoder (AE)-based methods can introduce modality bias, leading to fused images that favor one input over the other. This paper aims to overcome these limitations by developing a fusion technique that produces high-quality, detailed, and balanced fused images, and importantly, extends this capability to video fusion for applications like autonomous navigation.
Methods
DAE-Fuse employs a two-phase framework:
- Phase 1: Adversarial Feature Extraction: This phase focuses on enhancing the encoder's ability to extract features by reconstructing single-modality images. It uses a parallel Transformer-CNN architecture for shallow and deep feature extraction. An adversarial game is introduced with discriminative blocks to differentiate between input images and their reconstructions, ensuring more effective feature learning.
- Phase 2: Attention-Guided Cross-Modality Fusion: Here, a cross-attention module is used to capture complex correlations between encoded images from different modalities. The same discriminative blocks from Phase 1 are repurposed to distinguish between source images and the fused image. This adversarial learning encourages the fused result to evenly integrate information, preventing modality bias.
The framework also incorporates:
- Cross-modality attention: This module allows for interaction between feature embeddings of different modalities before fusion, addressing limitations of simple element-wise additions.
- Temporal Consistency Loss: For video fusion, a loss function is introduced to ensure smooth transitions between consecutive frames.
Datasets
The paper evaluates DAE-Fuse on several datasets:
- Infrared-Visible Image Fusion (IVIF): MSRS, RoadScene, and TNO datasets are used for standard IVIF benchmarks.
- Multi-Modality Object Detection (MMOD): The M³FD dataset is used to evaluate the object detection capabilities of the fused images.
- Medical Image Fusion: MRI-CT datasets are used to demonstrate the model's generalizability.
- Infrared-Visible Video Fusion: The RoadScene dataset is used for video fusion experiments.
Key Insights and Ideas
- Two-Phase Adversarial Learning: The separation into feature extraction and fusion phases, each with its own adversarial objectives, allows for more targeted improvements in both feature quality and fusion balance.
- Attention for Cross-Modality Interaction: The cross-attention module is crucial for enabling modalities to "understand" each other before fusion, leading to a more harmonious integration of information.
- Adaptive Fusion for Robustness: The adaptive nature of the discriminative blocks and adversarial training helps the model learn to balance modalities based on their complementary strengths, which is vital for challenging conditions.
- Generalizability Beyond IVIF: Demonstrating strong performance on medical image fusion highlights the framework's versatility and potential for broader applications.
Key Results
- State-of-the-Art Performance: DAE-Fuse achieves state-of-the-art results on multiple IVIF benchmarks (TNO, MSRS, RoadScene) across various quantitative metrics (EN, SD, SF, MI, SCD, VIF, Qabf).
- Superior Qualitative Results: Qualitative comparisons show DAE-Fuse producing sharper, more detailed, and less biased fused images compared to GAN-based and other AE-based methods.
- Improved Object Detection: Fused images from DAE-Fuse significantly improve object detection performance (mAP@50) on the M³FD dataset, successfully detecting objects missed by individual modalities.
- Effective Medical Image Fusion: The model generalizes well to MRI-CT fusion, outperforming other methods and producing balanced results.
- Ablation Studies: Experiments confirm the critical roles of the discriminative blocks and the cross-attention module in achieving DAE-Fuse's superior performance.
Limitations and Future Work
- Computational Cost: While not explicitly detailed as a limitation, the use of Transformer components and multi-phase adversarial training suggests potential for higher computational demands compared to simpler methods.
- Future Work: The paper implicitly suggests future work in exploring more advanced attention mechanisms, further optimizing the adversarial training process, and potentially extending the framework to other multi-modality fusion tasks beyond image and video. The authors also highlight the potential for real-world deployment in autonomous systems.