Degradation-Resistant Unfolding Network for Heterogeneous Image Fusion

Chunming He, Kai Li, Guoxia Xu, Yulun Zhang, Runze Hu, Zhenhua Guo, Xiu Li

Abstract

Heterogeneous Image Fusion (HIF) is essential for applications requiring multi-sensor data integration, such as autonomous driving and medical imaging. Traditional deep learning models often struggle with real-world degradations like noise, smoke, or extreme lighting. We present DeRUN (Degradation-Resistant Unfolding Network), a framework that combines the interpretability of mathematical optimization with the representational power of neural networks. By unfolding the ADMM algorithm and incorporating a novel directional entropy loss, DeRUN achieves state-of-the-art performance and robustness across natural, medical, and biological imaging domains.

Introduction

The primary goal of Heterogeneous Image Fusion (HIF) is to merge information from diverse sensors—typically visible-light and infrared cameras—to create a single, high-quality image. While visible light provides sharp textures, infrared sensors capture thermal signatures, which are invaluable in low-visibility conditions.

However, current HIF methods face two significant challenges. First, they are highly vulnerable to environmental degradations; smoke, fog, or sensor noise often lead to blurry or artifact-laden results. Second, many deep learning models operate as "black boxes," lacking the interpretability and mathematical grounding found in traditional optimization-based methods. While Deep Unfolding Networks (DUNs) attempt to bridge this gap, they are often computationally expensive and sensitive to complex noise.

Methodology: The DeRUN Framework

DeRUN addresses these challenges through a mathematically grounded, multi-stage architecture.

Heterogeneous Image Fusion Model (HIFM)

The foundation of DeRUN is a new mathematical model designed specifically for degradation resistance. Unlike standard models that assume clean inputs, HIFM prioritizes salient information while actively suppressing noise and artifacts.

Deep Unfolding with ADMM

The researchers "unfold" the iterative steps of the Alternating Direction Method of Multipliers (ADMM) into a neural network structure. This ensures that each layer of the network corresponds to a clear mathematical step, providing both interpretability and a structured path for optimization.

Key Innovations

  • Joint Constraint Strategy (JCS): This module integrates physical and denoising constraints into the structure-preservation layers, ensuring sharp edges even in degraded scenarios.
  • Partial Weight Sharing Module (PWSM): To optimize for speed and memory, DeRUN uses a feature extractor that shares weights across stages where appropriate, significantly reducing the model's computational footprint.
  • Local Gradient Directional Entropy (LGDE) Loss: A critical insight of this work is the use of gradient direction rather than just magnitude. While noise is typically random and lacks directional consistency, real image textures exhibit clear directional trends. LGDE allows the model to effectively "see through" noise by focusing on these consistent directional patterns.

Experimental Evaluation

DeRUN was rigorously evaluated across three domains: natural images (TNO, M3FD, LLVIP), medical scans (PET/MRI, SPECT/MRI), and biological imaging (GFP/Phase Contrast).

Performance and Robustness

The model consistently achieved top-tier results across standard metrics like SSIM and PSNR. Notably, DeRUN demonstrated exceptional stability in simulated harsh conditions, such as heavy rain and haze, where competing models experienced significant quality degradation.

Comparison of DeRUN against IVF-Net under degraded conditions (low light and smoke). DeRUN maintains superior clarity and detail preservation.

Computational Efficiency

Beyond image quality, DeRUN is highly efficient. It requires fewer floating-point operations (FLOPs) and offers faster processing times than previous unfolding networks, making it suitable for real-time deployment in edge devices like autonomous vehicle controllers.

Downstream Task Impact

To validate the practical utility of the fused images, the researchers tested them in object detection (YOLOv5) and tracking tasks. Images fused by DeRUN led to higher detection accuracy, proving that the network preserves the semantic information necessary for machine vision systems.

Conclusion

DeRUN represents a significant advancement in the field of image fusion. By shifting the focus from gradient magnitude to directional entropy and leveraging a mathematically grounded unfolding architecture, the model provides a robust solution for multi-sensor integration in challenging environments. Future work may explore extending this framework to additional sensor modalities like LiDAR.

Summary updated

Abstract

Heterogeneous image fusion (HIF) techniques aim to enhance image quality by merging complementary information from images captured by different sensors. Among these algorithms, deep unfolding network (DUN)-based methods achieve promising performance but still suffer from two issues: they lack a degradation-resistant-oriented fusion model and struggle to adequately consider the structural properties of DUNs, making them vulnerable to degradation scenarios. In this paper, we propose a Degradation-Resistant Unfolding Network (DeRUN) for the HIF task to generate high-quality fused images even in degradation scenarios. Specifically, we introduce a novel HIF model for degradation resistance and derive its optimization procedures. Then, we incorporate the optimization unfolding process into the proposed DeRUN for end-to-end training. To ensure the robustness and efficiency of DeRUN, we employ a joint constraint strategy and a lightweight partial weight sharing module. To train DeRUN, we further propose a gradient direction-based entropy loss with powerful texture representation capacity. Extensive experiments show that DeRUN significantly outperforms existing methods on four HIF tasks, as well as downstream applications, with cheaper computational and memory costs.
BibTeX
@inproceedings{he2023degradation,
  author = {Chunming He and Kai Li and Guoxia Xu and Yulun Zhang and Runze Hu and Zhenhua Guo and Xiu Li},
  title = {{Degradation-Resistant Unfolding Network for Heterogeneous Image Fusion}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01159},
}
Figure