Abstract
The recent Segment Anything Model (SAM) demonstrates strong instance segmentation performance across various downstream tasks. However, SAM is trained solely on RGB data, limiting its direct applicability to RGB-thermal (RGB-T) semantic segmentation. Given that RGB-T provides a robust solution for scene understanding in adverse weather and lighting conditions, such as low light and overexposure, we propose a novel framework, SARTM, which customizes the powerful SAM for RGB-T semantic segmentation. Our key idea is to unleash the potential of SAM while introducing semantic understanding modules for RGB-T data pairs. Specifically, our framework first involves fine-tuning the original SAM by adding extra LoRA layers, aiming at preserving SAM's strong generalization and segmentation capabilities for downstream tasks. Secondly, we introduce language information as guidance for training our SARTM. To address cross-modal inconsistencies, we introduce a Cross-Modal Knowledge Distillation (CMKD) module that effectively achieves modality adaptation while maintaining its generalization capabilities. This semantic module enables the minimization of modality gaps and alleviates semantic ambiguity, facilitating the combination of any modality under any visual conditions. Furthermore, we enhance the segmentation performance by adjusting the segmentation head of SAM and incorporating an auxiliary semantic segmentation head, which integrates multi-scale features for effective fusion. Extensive experiments are conducted across three multi-modal RGB-T semantic segmentation benchmarks: MFNET, PST900, and FMB. Both quantitative and qualitative results consistently demonstrate that the proposed SARTM significantly outperforms state-of-the-art approaches across a variety of conditions.
Introduction
Autonomous systems, particularly self-driving cars, rely heavily on semantic segmentation to understand their surroundings. While standard RGB cameras perform well in clear daylight, they struggle in "edge cases" like heavy fog, glare, or total darkness. Thermal (infrared) cameras provide a robust alternative by capturing heat signatures, but they lack the rich texture and color information of RGB.
The challenge lies in effective fusion. While the Segment Anything Model (SAM) has revolutionized RGB segmentation, it was trained exclusively on RGB data and designed for instance segmentation rather than semantic labeling. SARTM aims to bridge this gap, adapting the powerful SAM2 architecture for pixel-level semantic segmentation in the RGB-T domain.
Methodology
LoRA-based Adaptation
Rather than retraining the massive SAM2 image encoder from scratch—which would be computationally expensive and risk losing the model's general knowledge—SARTM utilizes Low-Rank Adaptation (LoRA). By adding extra LoRA layers to the frozen SAM2 image encoder, the model adapts to thermal data by training only a small set of auxiliary weights. This strategy preserves SAM's strong generalization while specializing it for thermal heat signatures.
Language-Aided Distillation
The core innovation of SARTM is the use of language as a "semantic anchor." The framework employs a pre-trained CLIP text encoder to generate semantic embeddings for class names (e.g., "person," "car," "tree"). A Cross-Modal Knowledge Distillation (CMKD) module aligns the visual features from RGB-T pairs with these language embeddings. Since a concept like a "car" remains consistent across modalities, language acts as a bridge to minimize the modality gap and reduce semantic ambiguity during fusion.
Multi-Scale Feature Fusion
To handle objects of varying sizes, the model generates three distinct feature maps: a Semantic Feature Map (SFM), a Fine-Grained Feature Pyramid (FFP), and an Intermediate-Resolution Feature Pyramid (IFP). These are processed through a dual-path prediction system consisting of an extended SAM2 mask decoder and an auxiliary semantic segmentation head. This combination ensures that the model captures both high-level semantics and precise object boundaries.
Experimental Results
Benchmarks and Performance
SARTM was evaluated on three major RGB-T benchmarks: MFNet (urban driving), PST900 (underground environments), and FMB (diverse weather conditions). The model achieved state-of-the-art (SOTA) results across all three:
- PST900: Reached 89.88% mIoU, outperforming previous leaders by 0.88%, with significant gains in identifying life-critical categories like "Survivor."
- MFNet: Achieved 60.03% mIoU, demonstrating superior performance in nighttime detection where RGB-only models typically fail.
- FMB: Reached 61.57% mIoU, proving robustness across 15 different object classes in fog and rain.
Qualitative Analysis
Visual comparisons highlight SARTM's ability to produce cleaner contours and fewer false positives in cluttered or low-light environments. As shown in the urban driving scenarios, SARTM accurately segments pedestrians and vehicles that are nearly invisible to standard RGB sensors.
Conclusion
SARTM successfully adapts the Segment Anything Model for the RGB-T domain by leveraging language-guided distillation and efficient LoRA fine-tuning. By using CLIP-derived text embeddings as a universal anchor, the model effectively bridges the gap between RGB and thermal modalities. Future work will explore extending this approach to other sensors, such as LiDAR, and developing dynamic fusion mechanisms to automatically weight modality importance based on environmental conditions.
Summary updated