Direction-aware multi-scale gradient loss for infrared and visible image fusion

Kaixuan Yang, Wei Xiang, Zhenshuai Chen, Tong Jin, Yunpeng LiuarXiv 2025 · DOI · arXiv

Abstract

Standard infrared and visible image fusion often suffers from blurry edges because training losses discard directional gradient information. This paper introduces a direction-aware, multi-scale gradient loss that supervises horizontal and vertical components separately, preventing "gradient cancellation" and ensuring fused images retain sharp, high-contrast textures. It is a "plug-and-play" loss function that improves existing fusion models without requiring any changes to their architecture.

The Problem: The "Cancellation" Trap

Infrared (IR) and visible (VIS) sensors capture different physical properties: IR detects thermal radiation, while VIS captures reflectance and fine textures. To combine these effectively, researchers use unsupervised loss functions to guide neural networks.

Most current methods rely on a gradient magnitude loss. However, by collapsing gradients into a single scalar magnitude, the directional vector information is lost. When IR and VIS images have opposing gradient signs at the same location—common at diagonal edges or complex textures—the signals can "cancel each other out" during training. This leads to "mutual dilution," resulting in fused images that appear muddy or lack sharp edge definition.

Proposed Method: Multi-scale Direction-Aligned Gradient Loss

The core innovation lies in constructing a target gradient that preserves directional integrity through three main components:

  1. Axis-wise Gating: Instead of a single magnitude, the method computes horizontal ($G_x$) and vertical ($G_y$) gradients separately using Sobel operators.
  2. Independent Selection: For every pixel, the loss function independently selects the $G_x$ and $G_y$ from either IR or VIS based on which has the larger absolute value, preserving the original sign. This allows the model to selectively integrate horizontal detail from one source and vertical structure from another at the same pixel.
  3. Multi-scale Supervision: To capture both fine-grained textures and coarse structural boundaries, the loss is applied across multiple image resolutions (1.0x, 0.5x, and 0.25x).
Comparison of standard gradient loss vs. the proposed axis-wise and multi-scale approach.

By keeping the sign of the gradient, the loss provides clearer guidance to the network, preventing the ambiguous supervision that plagues magnitude-only methods. Additionally, the authors utilize Mean Absolute Error (MAE) for optimization to increase robustness against outliers.

Experimental Evaluation

The researchers evaluated their approach across several standard benchmarks, including MSRS, FMB, M3FD, and the large-scale LLVIP dataset. Because this is purely a loss function, it was integrated into the existing ReCoNet fusion pipeline as a baseline to demonstrate its "plug-and-play" capability.

Key Results

  • Quantitative Performance: The method achieved superior scores across nearly all metrics, including Entropy (EN), Mutual Information (MI), and Standard Deviation (SD). On the MSRS dataset, it outperformed the next best method (TC-MoA) by approximately 0.4% to 2.3%.
  • Visual Clarity: Qualitatively, the fused images show significantly sharper pedestrians and more distinct edges in high-contrast regions. The "darkening" effect seen in other methods—caused by gradient cancellation—was effectively eliminated.
Qualitative comparison showing the superior edge definition and contrast of the proposed method (ours) against standard baselines.
  • Robustness: When trained on MSRS and tested on FMB and M3FD without fine-tuning, the model maintained its performance advantage, proving that the direction-aware logic generalizes well across different sensor types and lighting conditions.

Conclusion and Future Work

By decoupling gradient axes and preserving sign information, the proposed multi-scale gradient loss effectively solves the problem of gradient cancellation in image fusion. While the current work focuses on visual quality, future exploration could extend this loss to improve downstream machine-vision tasks, such as object detection or semantic segmentation in multi-sensor environments.

Summary updated

BibTeX
@misc{yang2025direction,
  author = {Kaixuan Yang and Wei Xiang and Zhenshuai Chen and Tong Jin and Yunpeng Liu},
  title = {{Direction-aware multi-scale gradient loss for infrared and visible image fusion}},
  year = {2025},
  eprint = {2510.13067},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.13067},
}
Figure