CSDNet: Detect Salient Object in Depth-Thermal via A Lightweight Cross Shallow and Deep Perception Network

Xiaotong Yu, Ruihan Xie, Zhihe Zhao, Chang Wen Chen

Abstract

While we enjoy the richness and informativeness of multimodal data, it also introduces interference and redundancy of information. To achieve optimal domain interpretation with limited resources, we propose CSDNet, a lightweight Cross Shallow and Deep Perception Network designed to integrate two modalities with less coherence, thereby discarding redundant information or even modality. We implement our CSDNet for Salient Object Detection (SOD) task in robotic perception. The proposed method capitalises on spatial information prescreening and implicit coherence navigation across shallow and deep layers of the depth-thermal (D-T) modality, prioritising integration over fusion to maximise the scene interpretation. To further refine the descriptive capabilities of the encoder for the less-known D-T modalities, we also propose SAMAEP to guide an effective feature mapping to the generalised feature space. Our approach is tested on the VDT-2048 dataset, leveraging the D-T modality outperforms those of SOTA methods using RGB-T or RGB-D modalities for the first time, achieves comparable performance with the RGB-D-T triple-modality benchmark method with 5.97 times faster at runtime and demanding 0.0036 times fewer FLOPs.

Introduction

In the world of robotic perception, the trend of stacking modalities—RGB, Depth, and Thermal—has improved performance at the cost of massive computational requirements, high latency, and privacy concerns. CSDNet proposes a strategic shift: instead of adding more sensors, we should integrate them more efficiently. By focusing on Depth (D) and Thermal (T) data, which exhibit low coherence and thus provide highly complementary information with minimal redundancy, CSDNet matches the accuracy of heavy triple-modality models while remaining light enough for edge devices like drones or mobile robots.

Methodology

CSDNet utilizes a lightweight MobileNet-V2 backbone enhanced by three novel components designed to maximize scene interpretation from D-T data:

  • CFAR Saliency Prescreening (CFARSP): Borrowing from radar technology, this module acts as a spatial filter in the shallow layers to identify significant features and suppress background noise before heavy processing begins.
  • Implicit Coherence Activation Navigation (ICAN): In deeper layers, ICAN uses "soft logic" operations (AND, OR, and XOR) to highlight consistent features and resolve discrepancies between the Depth and Thermal views.
  • SAM-Assist Encoder Pre-training (SAMAEP): To adapt backbones typically pre-trained on RGB, the Segment Anything Model (SAM) acts as a teacher during pre-training, guiding the encoder to map D-T data into a generalized feature space.
The CSDNet architecture featuring the CFARSP module for shallow feature prescreening and the ICAN module for deep feature navigation.

Experimental Results

The model was evaluated on the VDT-2048 dataset across 14 challenging robotic scenarios, including low illumination and thermal crossover. CSDNet achieved a Mean Absolute Error (MAE) of 0.0029, nearly identical to the much heavier triple-modality benchmark (0.0027).

Key efficiency gains include:

  • Speed: CSDNet runs at 10.27 FPS, roughly 5.97x faster than the triple-modality benchmark (HWSI).
  • Efficiency: It requires only 1.30G FLOPs, representing a reduction of over 99.6% compared to the benchmark's 357.93G.
t-SNE visualization of modality coherence and visual comparison demonstrating CSDNet's performance against the triple-modality HWSI benchmark.

Conclusion

CSDNet demonstrates that a Depth-Thermal dual-modality approach can outperform state-of-the-art RGB-T or RGB-D methods for the first time. By prioritizing integration over simple fusion and leveraging low-coherence modalities, the network provides a roadmap for high-performance, privacy-preserving robotic vision that is computationally feasible for real-time edge applications. Future work will focus on improving the resolution of feature mapping to better capture small salient objects.

Summary updated

Abstract

While we enjoy the richness and informativeness of multimodal data, it also introduces interference and redundancy of information. To achieve optimal domain interpretation with limited resources, we propose CSDNet, a lightweight \textbf{C}ross \textbf{S}hallow and \textbf{D}eep Perception \textbf{Net}work designed to integrate two modalities with less coherence, thereby discarding redundant information or even modality. We implement our CSDNet for Salient Object Detection (SOD) task in robotic perception. The proposed method capitalises on spatial information prescreening and implicit coherence navigation across shallow and deep layers of the depth-thermal (D-T) modality, prioritising integration over fusion to maximise the scene interpretation. To further refine the descriptive capabilities of the encoder for the less-known D-T modalities, we also propose SAMAEP to guide an effective feature mapping to the generalised feature space. Our approach is tested on the VDT-2048 dataset, leveraging the D-T modality outperforms those of SOTA methods using RGB-T or RGB-D modalities for the first time, achieves comparable performance with the RGB-D-T triple-modality benchmark method with 5.97 times faster at runtime and demanding 0.0036 times fewer FLOPs. Demonstrates the proposed CSDNet effectively integrates the information from the D-T modality. The code will be released upon acceptance.
BibTeX
@misc{yu2024csdnet,
  author = {Xiaotong Yu and Ruihan Xie and Zhihe Zhao and Chang Wen Chen},
  title = {{CSDNet: Detect Salient Object in Depth-Thermal via A Lightweight Cross Shallow and Deep Perception Network}},
  year = {2024},
  eprint = {2403.10104},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.10104},
}
Figure