Introduction
Autonomous robotic systems require robust perception across diverse environmental conditions, ranging from bright daylight to total darkness. While RGB cameras provide high-resolution semantic details in well-lit scenes, they struggle in low-light or adverse weather. Thermal sensors offer a critical complement by capturing heat signatures, yet effective RGB-Thermal (RGB-T) fusion remains challenging due to data scarcity and the inherent RGB bias of large-scale foundation models.
This paper introduces SHIFNet (SAM2-driven Hybrid Interactive Fusion Network), a framework designed to adapt the Segment Anything Model 2 (SAM2) for the RGB-T domain. By leveraging language as a modality-agnostic bridge, SHIFNet unlocks the generalization power of SAM2 for multi-modal robotic perception without requiring massive retraining.
Methodology: The SHIFNet Paradigm
SHIFNet maintains the core of the SAM2 architecture while introducing specialized modules for cross-modal interaction and semantic decoding. The framework is designed for efficiency, utilizing adapters within the frozen SAM2 encoder to achieve state-of-the-art performance with only 32.27M trainable parameters.
Semantic-Aware Cross-modal Fusion (SACF)
Traditional fusion methods often fail when one modality is significantly degraded. SACF addresses this by using linguistic guidance from a pre-trained Vision-Language Model (LanguageBind). Text embeddings act as a "modality arbiter," dynamically weighting RGB and thermal features based on their reliability for specific objects or lighting conditions.
Heterogeneous Prompting Decoder (HPD)
To overcome SAM2's "geometry-first" bias—where the model excels at finding boundaries but lacks semantic understanding of multi-modal data—the HPD replaces the standard mask decoder. It utilizes category embeddings (e.g., "pedestrian", "car") to guide segmentation. A key component is the Semantic Enhancement Module (SEM), which captures global context to ensure physical and semantic consistency in the final output.
Experimental Evaluation
SHIFNet was evaluated on three major RGB-T benchmarks: PST900 (subterranean), FMB (diverse weather), and MFNet (urban driving).
- State-of-the-Art Performance: SHIFNet achieved 89.8% mIoU on PST900 and 67.8% mIoU on FMB, significantly outperforming previous methods like HAPNet and MMSFormer.
- Robustness in Adverse Conditions: In safety-critical tasks like pedestrian detection on the MFNet dataset, the model demonstrated 76.5% accuracy. Qualitative results show that SHIFNet maintains complete silhouettes in pitch-black conditions by effectively leaning on the thermal modality guided by text prompts.
- Efficiency: With an average inference time of 123 ms on an NVIDIA RTX A6000, SHIFNet supports near-real-time perception for robotic platforms.
Conclusion
SHIFNet demonstrates that language guidance is a powerful tool for bridging the gap between visible and thermal modalities. By adapting SAM2 with semantic-aware fusion and decoding, the framework provides a robust, efficient solution for all-weather robotic vision. Future work will focus on expanding zero-shot capabilities and further optimizing the model for low-power edge devices.
Summary updated