Abstract
Multi-modal salient object detection (SOD) through the integration of additional data such as depth or thermal information has become a significant task in computer vision. Traditionally, the challenges of identifying salient objects in RGB, RGB-D (Depth), and RGB-T (Thermal) images are tackled separately, often leading to poorly defined object edges or overconfident inaccurate predictions. We propose DiMSOD, a novel approach that treats multi-modal SOD as a conditional mask generation task utilizing diffusion models. DiMSOD enables the concurrent use of local (depth/thermal maps) and global controls (images) within a unified model for progressive denoising. By fine-tuning a local control adapter on a pre-trained Stable Diffusion model, DiMSOD reduces training costs while enhancing multi-modal integration. Extensive experiments demonstrate that DiMSOD achieves superior performance across RGB, RGB-D, and RGB-T datasets compared to previous methods.
Introduction: Beyond Discriminative SOD
Salient Object Detection (SOD) aims to identify the most visually important objects in a scene. While standard RGB-based methods have improved, they often struggle in complex environments with low contrast or cluttered backgrounds. Multi-modal data, such as depth maps (RGB-D) or thermal images (RGB-T), provides critical auxiliary cues to overcome these challenges.
However, current "discriminative" models—which map an image directly to a mask in a single pass—suffer from two main issues: fusion friction and deterministic limits. Effectively combining different data types without creating blurry edges is difficult, and single-pass networks often struggle to refine boundaries in complex scenes. DiMSOD addresses these by treating SOD as a conditional mask generation task, using the iterative denoising power of diffusion models to progressively "carve out" the salient object from noise.
Methodology
The DiMSOD framework is built upon a pre-trained Latent Diffusion Model (Stable Diffusion), introducing several key architectural innovations to handle multi-modal guidance:
- Generative Paradigm: Instead of predicting a mask directly, the model starts with random noise and iteratively refines it into a saliency mask, guided by the input image and its auxiliary modality.
- SOD-ControlNet & FAN: This module handles local spatial controls. It uses a Feature Adaptive Network (FAN) to inject multi-modal data at multiple resolutions. FAN utilizes Instance Normalization to prevent the data statistics of different modalities (e.g., thermal vs. depth) from interfering with each other during training.
- Feature Injection Attention Network (FIAN): While ControlNet provides local guidance, FIAN provides global context. It uses a cross-attention mechanism to bridge the gap between the diffusion noise and high-level image features extracted by a Pyramid Vision Transformer (PVT) backbone.
Key Insights: The Power of Iterative Refinement
A central advantage of DiMSOD is its use of iterative refinement. The diffusion process acts as a built-in refinement module; each denoising step allows the model to correct discrepancies and sharpen edges. This leads to significantly better handling of intricate textures and small objects compared to "one-shot" discriminative models.
Furthermore, by leveraging Stable Diffusion as a backbone, DiMSOD inherits vast "objectness" knowledge learned from billions of images. This allows it to generalize better than models trained from scratch on smaller, specialized SOD datasets.
Results and Performance
DiMSOD consistently outperforms existing state-of-the-art (SOTA) methods, such as VSCode and LAFB, across all major metrics:
- RGB-D Performance: On the NJUD dataset, DiMSOD achieved a significant 6.48% improvement in Mean Absolute Error (MAE) over the second-best model.
- RGB Performance: On the DUTS-TE benchmark, it saw average improvements of ~1.05% in F-measure and 5% in MAE.
- Qualitative Superiority: Visual results show that DiMSOD is exceptionally good at maintaining structural completeness in "camouflaged" scenarios where the object color matches the background.
Conclusion and Future Directions
While DiMSOD sets a new benchmark for accuracy and boundary definition, its primary limitation is inference speed. Because diffusion models require multiple iterations (20 steps in this implementation) to generate a single mask, they are slower than traditional CNN or Transformer-based models. Future research will focus on utilizing consistency models or distillation techniques to reduce sampling steps without sacrificing the high-quality boundaries that define the diffusion approach.