Abstract
Heterogeneous Image Fusion (HIF) is essential for applications requiring multi-sensor data integration, such as autonomous driving and medical imaging. Traditional deep learning models often struggle with real-world degradations like noise, smoke, or extreme lighting. We present DeRUN (Degradation-Resistant Unfolding Network), a framework that combines the interpretability of mathematical optimization with the representational power of neural networks. By unfolding the ADMM algorithm and incorporating a novel directional entropy loss, DeRUN achieves state-of-the-art performance and robustness across natural, medical, and biological imaging domains.
Introduction
The primary goal of Heterogeneous Image Fusion (HIF) is to merge information from diverse sensors—typically visible-light and infrared cameras—to create a single, high-quality image. While visible light provides sharp textures, infrared sensors capture thermal signatures, which are invaluable in low-visibility conditions.
However, current HIF methods face two significant challenges. First, they are highly vulnerable to environmental degradations; smoke, fog, or sensor noise often lead to blurry or artifact-laden results. Second, many deep learning models operate as "black boxes," lacking the interpretability and mathematical grounding found in traditional optimization-based methods. While Deep Unfolding Networks (DUNs) attempt to bridge this gap, they are often computationally expensive and sensitive to complex noise.
Methodology: The DeRUN Framework
DeRUN addresses these challenges through a mathematically grounded, multi-stage architecture.
Heterogeneous Image Fusion Model (HIFM)
The foundation of DeRUN is a new mathematical model designed specifically for degradation resistance. Unlike standard models that assume clean inputs, HIFM prioritizes salient information while actively suppressing noise and artifacts.
Deep Unfolding with ADMM
The researchers "unfold" the iterative steps of the Alternating Direction Method of Multipliers (ADMM) into a neural network structure. This ensures that each layer of the network corresponds to a clear mathematical step, providing both interpretability and a structured path for optimization.
Key Innovations
- Joint Constraint Strategy (JCS): This module integrates physical and denoising constraints into the structure-preservation layers, ensuring sharp edges even in degraded scenarios.
- Partial Weight Sharing Module (PWSM): To optimize for speed and memory, DeRUN uses a feature extractor that shares weights across stages where appropriate, significantly reducing the model's computational footprint.
- Local Gradient Directional Entropy (LGDE) Loss: A critical insight of this work is the use of gradient direction rather than just magnitude. While noise is typically random and lacks directional consistency, real image textures exhibit clear directional trends. LGDE allows the model to effectively "see through" noise by focusing on these consistent directional patterns.
Experimental Evaluation
DeRUN was rigorously evaluated across three domains: natural images (TNO, M3FD, LLVIP), medical scans (PET/MRI, SPECT/MRI), and biological imaging (GFP/Phase Contrast).
Performance and Robustness
The model consistently achieved top-tier results across standard metrics like SSIM and PSNR. Notably, DeRUN demonstrated exceptional stability in simulated harsh conditions, such as heavy rain and haze, where competing models experienced significant quality degradation.
Computational Efficiency
Beyond image quality, DeRUN is highly efficient. It requires fewer floating-point operations (FLOPs) and offers faster processing times than previous unfolding networks, making it suitable for real-time deployment in edge devices like autonomous vehicle controllers.
Downstream Task Impact
To validate the practical utility of the fused images, the researchers tested them in object detection (YOLOv5) and tracking tasks. Images fused by DeRUN led to higher detection accuracy, proving that the network preserves the semantic information necessary for machine vision systems.
Conclusion
DeRUN represents a significant advancement in the field of image fusion. By shifting the focus from gradient magnitude to directional entropy and leveraging a mathematically grounded unfolding architecture, the model provides a robust solution for multi-sensor integration in challenging environments. Future work may explore extending this framework to additional sensor modalities like LiDAR.
Summary updated