Dual-modal Prior Semantic Guided Infrared and Visible Image Fusion for Intelligent Transportation System

Jing Li, Lu Bai, Bin Yang, Chang Li, Lingfei Ma, Lixin Cui, Edwin R. Hancock

Abstract

This paper introduces a novel dual-modal prior semantic guided approach for infrared (IR) and visible (VIS) image fusion, specifically optimized for Intelligent Transportation Systems (ITS). Traditional fusion methods often suffer from "task-coupling" and "semantic confusion" by using single-branch encoders that fail to account for the distinct domain characteristics of IR and VIS sensors. Our method employs parallel segmentation branches to independently extract significant semantics from each modality, coupled with a Refined Feature Adaptive-Modulation (RFaM) mechanism and a Multi-level Representation-Adaptive Fusion (MRaF) module. Experimental results on the MSRS and FMB datasets demonstrate that our approach significantly outperforms state-of-the-art methods in both visual quality and downstream semantic segmentation accuracy (mIoU), while maintaining high computational efficiency.

Introduction: The Task-Coupling Dilemma in ITS

In Intelligent Transportation Systems, sensors must operate reliably under adverse conditions such as nighttime, fog, or intense glare. While infrared (IR) sensors are excellent at detecting thermal signatures (e.g., pedestrians or active engines), they typically lack fine texture. Conversely, visible (VIS) sensors provide rich environmental details like lane markings and license plates but are highly susceptible to low-light conditions.

Comparison of input modalities (Visible and Infrared) alongside the segmentation Ground Truth.

Current Image Fusion (IVF) techniques attempt to merge these modalities into a single "super-image." However, they often fall into a "task-coupling" trap. Visual-centric methods prioritize human aesthetics at the expense of high-level semantic features required by AI models. Meanwhile, task-driven methods often utilize cascaded structures that force IR and VIS data through a shared feature space, leading to "semantic confusion" because the unique domain characteristics of each sensor are ignored.

Methodology: Dual-Modal Semantic Guidance

To address these challenges, the authors propose a three-branch architecture designed to preserve the independent strengths of each modality:

  • Parallel Semantic Branches: Instead of a shared encoder, the model uses two parallel branches to perform semantic segmentation on IR and VIS inputs independently. This allows the network to capture the "independent significant semantics" inherent to each sensor, avoiding the loss of unique prior information.
  • Refined Feature Adaptive-Modulation (RFaM): Integrated within the segmentation branches, this mechanism acts as an intelligent filter, perceiving and weighting features that are semantically distinct and useful for fusion.
  • Multi-level Representation-Adaptive Fusion (MRaF): The authors discovered that semantic information primarily resides in low-frequency components while textures are high-frequency. The MRaF module integrates these low-frequency "prior semantics" with high-frequency details from shallower layers, ensuring the final fused image is both semantically rich and visually sharp.
  • Hybrid Architecture: The model leverages both Transformers (for long-range global relations) and CNNs (for local inductive bias and textures) within each branch to optimize feature extraction.

Experimental Results

The proposed method was evaluated on two major public benchmarks: the MSRS (Multi-Spectral Road Scenarios) and FMB (Fusion Multi-Band) datasets.

Qualitative and quantitative comparison of our method against state-of-the-art models (TarDAL, PSFusion, SegMiF). The radar chart highlights superior performance across multiple metrics.

Quantitative Performance

On the MSRS dataset, the proposed method achieved a state-of-the-art mIoU of 68.86% for semantic segmentation tasks. It also led in fusion-specific metrics such as Mutual Information (MI) and Peak Signal-to-Noise Ratio (PSNR), indicating higher information retention and less noise compared to models like TarDAL, PSFusion, and SegMiF.

Qualitative Performance

Visually, the results show much sharper contours for vehicles and pedestrians. Notably, the method avoids the "halo" artifacts around bright lights (like street lamps) that often plague other fusion methods. This clarity is crucial for both human monitoring and automated detection.

Efficiency and Generalization

Despite its parallel architecture, the model is highly efficient, boasting fewer parameters and faster inference times (approx. 0.1s per image on an NVIDIA RTX 4090) than several recent competitors. Furthermore, the model maintained superior performance on the FMB dataset without additional fine-tuning, proving its robustness across different environmental conditions.

Conclusion

By leveraging dual-modal prior semantic guidance, this approach successfully bridges the gap between visual image fusion and high-level computer vision tasks. The use of parallel branches prevents semantic confusion, resulting in fused images that are optimized for both human perception and machine understanding in complex transportation environments. Future work will focus on real-time deployment on edge hardware and extending this guidance strategy to other tasks like object tracking or 3D scene reconstruction.

Summary updated

BibTeX
@article{li2025dual,
  author = {Jing Li and Lu Bai and Bin Yang and Chang Li and Lingfei Ma and Lixin Cui and Edwin R. Hancock},
  title = {{Dual-modal Prior Semantic Guided Infrared and Visible Image Fusion for Intelligent Transportation System}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2025},
  doi = {10.1109/tits.2025.3562267},
}
Figure