Dual-Modal Prior Semantic Guided Infrared and Visible Image Fusion for Intelligent Transportation System

Jing Li, Lu Bai, Bin Yang, Chang Li, Lingfei Ma, Lixin Cui, Edwin R. Hancock

TL;DR

This paper introduces a novel dual-modal prior semantic guided image fusion method designed to optimize both visual quality and high-level scene understanding for Intelligent Transportation Systems (ITS). By decoupling semantic segmentation and image fusion into parallel, modality-specific branches and utilizing a frequency-aware fusion mechanism, the network achieves state-of-the-art performance in both tasks while maintaining a remarkably lightweight parameter footprint.

What Problem Does This Paper Address?

Intelligent Transportation Systems (ITS) rely heavily on robust environmental perception. Infrared (IR) sensors excel at highlighting thermal targets (like pedestrians and vehicles) in low-light or harsh weather conditions, but they lack fine texture details. Conversely, visible (VIS) sensors capture rich textures and environmental cues (like road boundaries and license plates) but fail in poor lighting.

Infrared and Visible Image Fusion (IVF) aims to combine these modalities into a single, highly informative image. However, traditional IVF methods focus purely on making images look good to the human eye (pixel reconstruction), ignoring how the fused image performs in downstream AI tasks like semantic segmentation. Recent "task-driven" methods attempt to bridge this gap using cascaded architectures, but they often force the network to compromise on a single set of features, diluting the unique semantic properties of each sensor modality and introducing redundant information.

Methods

To overcome the limitations of cascaded architectures, the authors propose a parallel, dual-modality framework that treats semantic extraction and image fusion independently before intelligently combining them.

The overall framework of the proposed dual-modality semantic guided image fusion method.

Key components include:

  • Parallel Semantic Segmentation Branches: Instead of a single shared network, the architecture deploys two parallel branches—one for IR and one for VIS. Each branch uniquely combines a Transformer (to capture global, long-range dependencies) and a CNN (to capture local, fine-grained features).
  • Refined Feature Adaptive-Modulation (RFaM): Integrated within the segmentation branches, RFaM dynamically weights multi-scale features to identify and extract "significant prior semantics"—the features most critical for distinct semantic understanding.
  • Multi-level Representation-Adaptive Fusion (MRaF): By analyzing the frequency response of the network's layers, the authors discovered that critical semantic information resides primarily in the low-frequency spectrum, while visual details reside in high-frequency shallow features. The MRaF module explicitly blends these low-frequency semantic priors with high-frequency details using dynamically updated weights.
The architecture of the Refined Feature Adaptive-Modulation (RFaM) module.

Datasets

The model was trained and evaluated on two prominent public benchmarks that feature aligned infrared/visible image pairs and annotated semantic segmentation labels:

  • MSRS (Multi-Spectral Road Scenarios): Used for primary training, evaluation, and comparative analysis.
  • FMB (Functional Multi-spectral Benchmark): Used to evaluate the generalization capabilities of the model across diverse environmental conditions.

Key Insights and Ideas

  • Task Decoupling Prevents Feature Dilution: Forcing a single network cascade to optimize for both raw pixel reconstruction and high-level segmentation leads to poor feature representation. Separating them into parallel semantic branches preserves the independent modal integrity of IR and VIS sensors.
  • Frequency-Aware Fusion Matters: Blindly aggregating deep and shallow features can introduce noise. Realizing that semantics are inherently low-frequency and textures are high-frequency allowed the authors to design a targeted fusion strategy (MRaF) that yields cleaner, more natural-looking results without losing contextual AI performance.

Key Results

The proposed method demonstrates clear superiority over existing state-of-the-art task-driven methods (such as PSFusion, SegMiF, and TarDAL) across both quantitative metrics and qualitative visual inspection.

Comparison of the proposed method against state-of-the-art task-driven fusion methods.
  • Semantic Segmentation Accuracy: On the MSRS dataset, the method achieved a top-tier mean Intersection over Union (mIoU) of 68.86% using a ViT-Adapter backbone and 67.50% using a Segformer backbone, outperforming all competitors.
  • Image Fusion Quality: The framework achieved the highest scores in Mutual Information (MI = 2.85), Structural Similarity (SSIM = 1.19), and Peak Signal-to-Noise Ratio (PSNR = 16.98) on the MSRS dataset, indicating superior information retention and structural fidelity.
  • Efficiency: Despite utilizing parallel branches, the targeted feature extraction of the RFaM module keeps the network lightweight. It uses just 10.32M parameters—nearly a quarter of the size of PSFusion (43.83M) and SegMiF (43.53M)—and clocks the fastest inference time at 0.1 seconds.

Limitations and Future Work

Although highly efficient compared to other semantic-driven fusion models, the framework still carries a larger parameter footprint than purely vision-perception-oriented models or simpler object-detection networks (like TarDAL at 0.28M) due to its dual-branch segmentation architecture. The authors suggest that future work will focus on further compressing these parallel branches and extending this prior semantic guidance framework to real-time video object tracking within complex ITS environments.

Summary updated

Abstract

Infrared and visible image fusion (IVF) plays an important role in intelligent transportation system (ITS). The early works predominantly focus on boosting the visual appeal of the fused result, and only several recent approaches have tried to combine the high-level vision task with IVF. However, they prioritize the design of cascaded structure to seek unified suitable features and fit different tasks. Thus, they tend to typically bias toward to reconstructing raw pixels without considering the significance of semantic features. Therefore, we propose a novel prior semantic guided image fusion method based on the dual-modality strategy, improving the performance of IVF in ITS. Specifically, to explore the independent significant semantic of each modality, we first design two parallel semantic segmentation branches with a refined feature adaptive-modulation (RFaM) mechanism. RFaM can perceive the features that are semantically distinct enough in each semantic segmentation branch. Then, two pilot experiments based on the two branches are conducted to capture the significant prior semantic of two images, which then is applied to guide the fusion task in the integration of semantic segmentation branches and fusion branches. In addition, to aggregate both high-level semantics and impressive visual effects, we further investigate the frequency response of the prior semantics, and propose a multi-level representation-adaptive fusion (MRaF) module to explicitly integrate the low-frequent prior semantic with the high-frequent details. Extensive experiments on two public datasets demonstrate the superiority of our method over the state-of-the-art image fusion approaches, in terms of either the visual appeal or the high-level semantics.
BibTeX
@article{li2025dual,
  author = {Jing Li and Lu Bai and Bin Yang and Chang Li and Lingfei Ma and Lixin Cui and Edwin R. Hancock},
  title = {{Dual-Modal Prior Semantic Guided Infrared and Visible Image Fusion for Intelligent Transportation System}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2025},
  doi = {10.1109/tits.2025.3562267},
}
Figure