Abstract
While we enjoy the richness and informativeness of multimodal data, it also introduces interference and redundancy of information. To achieve optimal domain interpretation with limited resources, we propose CSDNet, a lightweight Cross Shallow and Deep Perception Network designed to integrate two modalities with less coherence, thereby discarding redundant information or even modality. We implement our CSDNet for Salient Object Detection (SOD) task in robotic perception. The proposed method capitalises on spatial information prescreening and implicit coherence navigation across shallow and deep layers of the depth-thermal (D-T) modality, prioritising integration over fusion to maximise the scene interpretation. To further refine the descriptive capabilities of the encoder for the less-known D-T modalities, we also propose SAMAEP to guide an effective feature mapping to the generalised feature space. Our approach is tested on the VDT-2048 dataset, leveraging the D-T modality outperforms those of SOTA methods using RGB-T or RGB-D modalities for the first time, achieves comparable performance with the RGB-D-T triple-modality benchmark method with 5.97 times faster at runtime and demanding 0.0036 times fewer FLOPs.
Introduction
In the world of robotic perception, the trend of stacking modalities—RGB, Depth, and Thermal—has improved performance at the cost of massive computational requirements, high latency, and privacy concerns. CSDNet proposes a strategic shift: instead of adding more sensors, we should integrate them more efficiently. By focusing on Depth (D) and Thermal (T) data, which exhibit low coherence and thus provide highly complementary information with minimal redundancy, CSDNet matches the accuracy of heavy triple-modality models while remaining light enough for edge devices like drones or mobile robots.
Methodology
CSDNet utilizes a lightweight MobileNet-V2 backbone enhanced by three novel components designed to maximize scene interpretation from D-T data:
- CFAR Saliency Prescreening (CFARSP): Borrowing from radar technology, this module acts as a spatial filter in the shallow layers to identify significant features and suppress background noise before heavy processing begins.
- Implicit Coherence Activation Navigation (ICAN): In deeper layers, ICAN uses "soft logic" operations (AND, OR, and XOR) to highlight consistent features and resolve discrepancies between the Depth and Thermal views.
- SAM-Assist Encoder Pre-training (SAMAEP): To adapt backbones typically pre-trained on RGB, the Segment Anything Model (SAM) acts as a teacher during pre-training, guiding the encoder to map D-T data into a generalized feature space.
Experimental Results
The model was evaluated on the VDT-2048 dataset across 14 challenging robotic scenarios, including low illumination and thermal crossover. CSDNet achieved a Mean Absolute Error (MAE) of 0.0029, nearly identical to the much heavier triple-modality benchmark (0.0027).
Key efficiency gains include:
- Speed: CSDNet runs at 10.27 FPS, roughly 5.97x faster than the triple-modality benchmark (HWSI).
- Efficiency: It requires only 1.30G FLOPs, representing a reduction of over 99.6% compared to the benchmark's 357.93G.
Conclusion
CSDNet demonstrates that a Depth-Thermal dual-modality approach can outperform state-of-the-art RGB-T or RGB-D methods for the first time. By prioritizing integration over simple fusion and leveraging low-coherence modalities, the network provides a roadmap for high-performance, privacy-preserving robotic vision that is computationally feasible for real-time edge applications. Future work will focus on improving the resolution of feature mapping to better capture small salient objects.
Summary updated