Infrared and Visible Image Fusion with Language-Driven Loss in CLIP Embedding Space

Yuhao Wang, Lingjuan Miao, Zhiqiang Zhou, Lei Zhang, Yajun Qiao

Abstract

Infrared-visible image fusion (IVIF) has attracted much attention owing to the highly-complementary properties of the two image modalities. Due to the lack of ground-truth fused images, the fusion output of current deep-learning based methods heavily depends on the loss functions defined mathematically. As it is hard to well mathematically define the fused image without ground truth, the performance of existing fusion methods is limited. In this paper, we propose to use natural language to express the objective of IVIF, which can avoid the explicit mathematical modeling of fusion output in current losses, and make full use of the advantage of language expression to improve the fusion performance. For this purpose, we present a comprehensive language-expressed fusion objective, and encode relevant texts into the multi-modal embedding space using CLIP. A language-driven fusion model is then constructed in the embedding space, by establishing the relationship among the embedded vectors representing the fusion objective and input image modalities. Finally, a language-driven loss is derived to make the actual IVIF aligned with the embedded language-driven fusion model via supervised training. Experiments show that our method can obtain much better fusion results than existing techniques.

1. Introduction

Infrared-visible image fusion (IVIF) aims to combine the thermal signatures of infrared (IR) images with the high-resolution textures of visible images. This is critical for applications ranging from autonomous driving to nighttime surveillance. However, a fundamental challenge persists: there is no "ground truth" for what a perfect fused image looks like.

Because researchers lack a target image for comparison, they typically rely on complex, hand-crafted mathematical loss functions based on structural similarity (SSIM), pixel intensity, or adversarial training. Unfortunately, rigid mathematical formulas often fail to capture subjective human concepts like "salient objects" or "clear background." Consequently, current models frequently struggle with artifacts, loss of detail, or poor contrast. This paper introduces LDFusion, a framework that replaces these rigid mathematical constraints with natural language descriptions optimized within the CLIP embedding space.

2. Methodology

LDFusion shifts the optimization process from pixel-space mathematics to the multi-modal embedding space of CLIP (Contrastive Language-Image Pre-training). The framework consists of several key components:

2.1 Language-Expressed Fusion Objective (LEFO)

Instead of a formula, the target is defined by a text prompt: "a clear image with detailed background and salient objects." This allows the model to prioritize human-perceptible qualities that are difficult to model mathematically.

2.2 CLIP-Driven Alignment and Directional Loss

The model leverages a pre-trained CLIP model to encode source images (IR and Visible) and text prompts into a shared embedding space. The core innovation is the Fusion Direction Loss ($L_d$), which utilizes "vector arithmetic" in CLIP space. The model calculates the directional vector from the text prompt "an infrared image" to the target fusion prompt. It then trains the fusion network to ensure the actual image transformation follows this same directional vector.

2.3 Regularization and Artifact Suppression

To prevent modality bias, a regularization term ensures the fused image maintains a balanced "angle" between both source modalities. Additionally, to combat CLIP's tendency to "hallucinate" textures in flat areas, the authors implemented a Patch Filter based on information entropy. This filter ignores low-information patches during the language-driven training phase, significantly reducing artifacts.

3. Experimental Evaluation

The model was evaluated on three standard IVIF benchmarks: M3FD, RoadScene, and TNO. LDFusion was compared against 11 state-of-the-art methods, including SwinFusion, TarDAL, and Text-IF.

3.1 Qualitative and Quantitative Results

Quantitatively, LDFusion achieved the highest scores across nearly all metrics, including Information Entropy (EN), Standard Deviation (SD), and Visual Information Fidelity (VIFF). Qualitatively, the results demonstrate superior contrast for thermal targets (such as pedestrians) while maintaining sharper background textures compared to previous methods.

Qualitative comparison of LDFusion against state-of-the-art methods (TIMFusion, Text-IF, DCEvo) on night street and tunnel scenes.

3.2 Downstream Task Performance

To validate the utility of the fusion, the outputs were processed by a YOLOv8 object detector. LDFusion provided the best Mean Average Precision (mAP), proving that the "salient objects" prompt effectively enhances machine vision performance in addition to human visual quality.

4. Conclusion and Future Work

LDFusion demonstrates that natural language is a powerful, flexible alternative to hand-crafted loss functions in image fusion. By leveraging the semantic richness of CLIP, the model achieves state-of-the-art results that align more closely with human perception. While the entropy-based patch filter mitigates artifacts, future work may involve fine-tuning CLIP specifically for non-visible spectra to further refine multi-modal alignment and reduce sensitivity to prompt engineering.

Summary updated

Abstract

Infrared-visible image fusion (IVIF) has attracted much attention owing to the highly-complementary properties of the two image modalities. Due to the lack of ground-truth fused images, the fusion output of current deep-learning based methods heavily depends on the loss functions defined mathematically. As it is hard to well mathematically define the fused image without ground truth, the performance of existing fusion methods is limited. In this paper, we propose to use natural language to express the objective of IVIF, which can avoid the explicit mathematical modeling of fusion output in current losses, and make full use of the advantage of language expression to improve the fusion performance. For this purpose, we present a comprehensive language-expressed fusion objective, and encode relevant texts into the multi-modal embedding space using CLIP. A language-driven fusion model is then constructed in the embedding space, by establishing the relationship among the embedded vectors representing the fusion objective and input image modalities. Finally, a language-driven loss is derived to make the actual IVIF aligned with the embedded language-driven fusion model via supervised training. Experiments show that our method can obtain much better fusion results than existing techniques. The code is available at https://github.com/wyhlaowang/LDFusion.
BibTeX
@misc{wang2024infrared,
  author = {Yuhao Wang and Lingjuan Miao and Zhiqiang Zhou and Lei Zhang and Yajun Qiao},
  title = {{Infrared and Visible Image Fusion with Language-Driven Loss in CLIP Embedding Space}},
  year = {2024},
  eprint = {2402.16267},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.16267},
}
Figure