Abstract
In the field of unmanned systems, such as autonomous vehicles and drones, the ability to accurately perceive environments through multi-modal sensors is critical. This paper introduces IRFS, an interactively reinforced multi-task framework designed to bridge the gap between low-level Infrared-Visible Image Fusion (IVIF) and high-level Saliency Object Detection (SOD). By treating fusion and detection as a single, mutually reinforcing loop, IRFS produces fused images with superior semantic clarity while achieving state-of-the-art accuracy in locating salient objects.
Introduction
Infrared sensors are highly effective at capturing heat signatures, making them ideal for identifying salient objects in low-light or cluttered environments. Conversely, visible-light sensors provide the rich textures and spatial details necessary for human and machine interpretation. Traditionally, these two modalities are combined via IVIF to create a single composite image. However, conventional fusion methods often prioritize visual aesthetics for human observers, frequently discarding the subtle semantic cues essential for downstream AI tasks like SOD.
Furthermore, many existing SOD models are computationally expensive and struggle to efficiently integrate multi-sensor data. This research addresses the disconnect between these two domains, proposing a paradigm where image fusion and object detection work in tandem rather than in isolation.
Methodology
The IRFS framework consists of two primary components linked by a novel training strategy:
1. FSFNet: Feature Screening-based Fusion
Unlike standard fusion networks that attempt to preserve all source information, the Feature Screening-based Fusion subnetwork (FSFNet) utilizes a specialized screening module. This module intentionally filters out background interference and prioritizes features most relevant to identifying salient objects. This ensures that the resulting fused image is optimized for machine perception.
2. FGC2Net: Fusion-Guided Cross-Complementary SOD
The Fusion-Guided Cross-Complementary SOD subnetwork (FGC2Net) serves as the detection engine. A key innovation here is the treatment of the fused image from FSFNet as a third modality. By using the fused image to guide the raw infrared and visible features, the network can better suppress noise and highlight the target object with higher precision.
3. Interactive Loop Learning
The core of the IRFS paradigm is its bidirectional optimization strategy. Rather than training the fusion and detection components separately, the authors employ an interactive loop:
- Semantic Feedback: The SOD task provides a "semantic loss" back to the fusion network, forcing it to generate images that are more "detectable."
- Input Optimization: The fusion network provides refined, high-quality inputs to the SOD task. This loop allows both tasks to reach an optimal balance, improving the performance of both the fusion and the detection outputs.
Experimental Evaluation
Datasets
The researchers evaluated IRFS across several benchmark datasets:
- VT5000: The primary dataset for joint evaluation, featuring 5,000 pairs of infrared/visible images with binary ground-truth masks.
- SOD Generalization: VT1000 and VT821 were used to test detection robustness on unseen data.
- Fusion Generalization: TNO, RoadScene, and M3FD were utilized to evaluate visual quality against other state-of-the-art (SOTA) methods.
Key Results
- Fusion Quality: IRFS demonstrated a 45.8% improvement in the Visual Information Fidelity (VIF) metric compared to existing methods, indicating that the fused images are significantly more representative of the source data.
- Detection Accuracy: On the VT5000 dataset, the model achieved a mean $F_\beta$ score of 0.835 and a Mean Absolute Error (MAE) of 0.034, outperforming ten specialized SOD methods.
- Efficiency: The model is remarkably lightweight; the fusion subnetwork occupies only 0.06Mb, and the system utilizes a siamese encoder based on ResNet-34 to achieve fast inference times suitable for real-time applications.
Discussion and Insights
The success of IRFS stems from two primary intuitions. First, the "Third Modality" approach provides the model with a pre-processed "map" that helps align features between disparate sensors. Second, the framework reframes image fusion as a semantic "adapter" that prepares raw data for high-level machine vision tasks. Ablation studies confirmed that the interactive loop is essential; when trained with SOD feedback, the fusion network produced images that were significantly sharper and more object-focused than those trained in isolation.
Conclusion and Future Work
IRFS represents a significant step toward integrating low-level and high-level vision tasks for multi-modal sensing. While the model shows state-of-the-art performance, future research will focus on enhancing robustness in extreme edge cases, such as heavy fog or total sensor failure. Additionally, the authors aim to expand this interactively reinforced logic to other high-level tasks, including semantic segmentation and instance tracking, to further empower autonomous systems in complex environments.
Summary updated