TL;DR
This paper introduces a novel dual-modal prior semantic guided image fusion method designed to optimize both visual quality and high-level scene understanding for Intelligent Transportation Systems (ITS). By decoupling semantic segmentation and image fusion into parallel, modality-specific branches and utilizing a frequency-aware fusion mechanism, the network achieves state-of-the-art performance in both tasks while maintaining a remarkably lightweight parameter footprint.
What Problem Does This Paper Address?
Intelligent Transportation Systems (ITS) rely heavily on robust environmental perception. Infrared (IR) sensors excel at highlighting thermal targets (like pedestrians and vehicles) in low-light or harsh weather conditions, but they lack fine texture details. Conversely, visible (VIS) sensors capture rich textures and environmental cues (like road boundaries and license plates) but fail in poor lighting.
Infrared and Visible Image Fusion (IVF) aims to combine these modalities into a single, highly informative image. However, traditional IVF methods focus purely on making images look good to the human eye (pixel reconstruction), ignoring how the fused image performs in downstream AI tasks like semantic segmentation. Recent "task-driven" methods attempt to bridge this gap using cascaded architectures, but they often force the network to compromise on a single set of features, diluting the unique semantic properties of each sensor modality and introducing redundant information.
Methods
To overcome the limitations of cascaded architectures, the authors propose a parallel, dual-modality framework that treats semantic extraction and image fusion independently before intelligently combining them.
Key components include:
- Parallel Semantic Segmentation Branches: Instead of a single shared network, the architecture deploys two parallel branches—one for IR and one for VIS. Each branch uniquely combines a Transformer (to capture global, long-range dependencies) and a CNN (to capture local, fine-grained features).
- Refined Feature Adaptive-Modulation (RFaM): Integrated within the segmentation branches, RFaM dynamically weights multi-scale features to identify and extract "significant prior semantics"—the features most critical for distinct semantic understanding.
- Multi-level Representation-Adaptive Fusion (MRaF): By analyzing the frequency response of the network's layers, the authors discovered that critical semantic information resides primarily in the low-frequency spectrum, while visual details reside in high-frequency shallow features. The MRaF module explicitly blends these low-frequency semantic priors with high-frequency details using dynamically updated weights.
Datasets
The model was trained and evaluated on two prominent public benchmarks that feature aligned infrared/visible image pairs and annotated semantic segmentation labels:
- MSRS (Multi-Spectral Road Scenarios): Used for primary training, evaluation, and comparative analysis.
- FMB (Functional Multi-spectral Benchmark): Used to evaluate the generalization capabilities of the model across diverse environmental conditions.
Key Insights and Ideas
- Task Decoupling Prevents Feature Dilution: Forcing a single network cascade to optimize for both raw pixel reconstruction and high-level segmentation leads to poor feature representation. Separating them into parallel semantic branches preserves the independent modal integrity of IR and VIS sensors.
- Frequency-Aware Fusion Matters: Blindly aggregating deep and shallow features can introduce noise. Realizing that semantics are inherently low-frequency and textures are high-frequency allowed the authors to design a targeted fusion strategy (MRaF) that yields cleaner, more natural-looking results without losing contextual AI performance.
Key Results
The proposed method demonstrates clear superiority over existing state-of-the-art task-driven methods (such as PSFusion, SegMiF, and TarDAL) across both quantitative metrics and qualitative visual inspection.
- Semantic Segmentation Accuracy: On the MSRS dataset, the method achieved a top-tier mean Intersection over Union (mIoU) of 68.86% using a ViT-Adapter backbone and 67.50% using a Segformer backbone, outperforming all competitors.
- Image Fusion Quality: The framework achieved the highest scores in Mutual Information (MI = 2.85), Structural Similarity (SSIM = 1.19), and Peak Signal-to-Noise Ratio (PSNR = 16.98) on the MSRS dataset, indicating superior information retention and structural fidelity.
- Efficiency: Despite utilizing parallel branches, the targeted feature extraction of the RFaM module keeps the network lightweight. It uses just 10.32M parameters—nearly a quarter of the size of PSFusion (43.83M) and SegMiF (43.53M)—and clocks the fastest inference time at 0.1 seconds.
Limitations and Future Work
Although highly efficient compared to other semantic-driven fusion models, the framework still carries a larger parameter footprint than purely vision-perception-oriented models or simpler object-detection networks (like TarDAL at 0.28M) due to its dual-branch segmentation architecture. The authors suggest that future work will focus on further compressing these parallel branches and extending this prior semantic guidance framework to real-time video object tracking within complex ITS environments.
Summary updated