Abstract
Standard infrared and visible image fusion often suffers from blurry edges because training losses discard directional gradient information. This paper introduces a direction-aware, multi-scale gradient loss that supervises horizontal and vertical components separately, preventing "gradient cancellation" and ensuring fused images retain sharp, high-contrast textures. It is a "plug-and-play" loss function that improves existing fusion models without requiring any changes to their architecture.
The Problem: The "Cancellation" Trap
Infrared (IR) and visible (VIS) sensors capture different physical properties: IR detects thermal radiation, while VIS captures reflectance and fine textures. To combine these effectively, researchers use unsupervised loss functions to guide neural networks.
Most current methods rely on a gradient magnitude loss. However, by collapsing gradients into a single scalar magnitude, the directional vector information is lost. When IR and VIS images have opposing gradient signs at the same location—common at diagonal edges or complex textures—the signals can "cancel each other out" during training. This leads to "mutual dilution," resulting in fused images that appear muddy or lack sharp edge definition.
Proposed Method: Multi-scale Direction-Aligned Gradient Loss
The core innovation lies in constructing a target gradient that preserves directional integrity through three main components:
- Axis-wise Gating: Instead of a single magnitude, the method computes horizontal ($G_x$) and vertical ($G_y$) gradients separately using Sobel operators.
- Independent Selection: For every pixel, the loss function independently selects the $G_x$ and $G_y$ from either IR or VIS based on which has the larger absolute value, preserving the original sign. This allows the model to selectively integrate horizontal detail from one source and vertical structure from another at the same pixel.
- Multi-scale Supervision: To capture both fine-grained textures and coarse structural boundaries, the loss is applied across multiple image resolutions (1.0x, 0.5x, and 0.25x).
By keeping the sign of the gradient, the loss provides clearer guidance to the network, preventing the ambiguous supervision that plagues magnitude-only methods. Additionally, the authors utilize Mean Absolute Error (MAE) for optimization to increase robustness against outliers.
Experimental Evaluation
The researchers evaluated their approach across several standard benchmarks, including MSRS, FMB, M3FD, and the large-scale LLVIP dataset. Because this is purely a loss function, it was integrated into the existing ReCoNet fusion pipeline as a baseline to demonstrate its "plug-and-play" capability.
Key Results
- Quantitative Performance: The method achieved superior scores across nearly all metrics, including Entropy (EN), Mutual Information (MI), and Standard Deviation (SD). On the MSRS dataset, it outperformed the next best method (TC-MoA) by approximately 0.4% to 2.3%.
- Visual Clarity: Qualitatively, the fused images show significantly sharper pedestrians and more distinct edges in high-contrast regions. The "darkening" effect seen in other methods—caused by gradient cancellation—was effectively eliminated.
- Robustness: When trained on MSRS and tested on FMB and M3FD without fine-tuning, the model maintained its performance advantage, proving that the direction-aware logic generalizes well across different sensor types and lighting conditions.
Conclusion and Future Work
By decoupling gradient axes and preserving sign information, the proposed multi-scale gradient loss effectively solves the problem of gradient cancellation in image fusion. While the current work focuses on visual quality, future exploration could extend this loss to improve downstream machine-vision tasks, such as object detection or semantic segmentation in multi-sensor environments.
Summary updated