Abstract
Salient Object Detection (SOD) is a fundamental computer vision task that identifies the most prominent objects in a scene. While RGB-based methods have achieved high performance, they often struggle in challenging environments characterized by low contrast, deep shadows, or extreme illumination. This paper introduces a robust multi-spectral SOD approach that integrates Near-Infrared (NIR) data with standard RGB images. To overcome the lack of large-scale labeled multi-spectral datasets, the authors propose an adversarial domain adaptation framework using CycleGAN to generate "pseudo-NIR" data. The approach demonstrates significant robustness in challenging lighting and low-contrast conditions, outperforming RGB-only baselines.
Introduction
Deep learning has significantly advanced RGB-based Salient Object Detection. However, these models frequently fail in complex real-world scenarios where foreground objects are camouflaged or obscured by lighting conditions. Near-Infrared (NIR) imaging offers a solution, as it captures physical material properties that remain visible even when color information is suppressed by shadows or glare.
The primary hurdle in multi-spectral SOD is the scarcity of synchronized, labeled RGB-NIR datasets. While RGB datasets like MSRA-B contain thousands of labeled images, multi-spectral counterparts are rare. This work bridges this gap through synthetic data generation and domain adaptation.
Methodology
The proposed framework treats multi-spectral SOD as a domain adaptation problem, transferring knowledge from a labeled "Source" domain (RGB) to an unlabeled or sparsely labeled "Target" domain (RGB-NIR).
1. Pseudo-NIR Generation
To leverage existing large-scale RGB datasets, the authors utilize a CycleGAN to learn the mapping between RGB and NIR modalities. This allows them to "hallucinate" NIR counterparts for standard RGB images, creating a synthetic multi-spectral training set.
2. Two-Branch SOD Network
The core architecture is a generator ($G$) based on a Fully Convolutional Network (FCN). It features two branches—one for RGB and one for NIR—that share weights. The features from both branches are fused to produce the final saliency map.
3. Adversarial Domain Adaptation
To ensure the model generalizes to real NIR sensors, a Discriminator ($D$) is introduced. The Discriminator attempts to distinguish between saliency maps generated from synthetic (source) data and real (target) data. The Generator is trained to produce domain-invariant features that "fool" the Discriminator, ensuring robust performance across both domains.
The Multi-spectral SOD Dataset
A key contribution of this work is the release of a new benchmark dataset consisting of 780 synchronized RGB and NIR image pairs. The dataset is meticulously annotated with pixel-level ground truth and categorized into four challenge types: Normal, Strong Light, Shadow, and Similar Appearance.
Results and Discussion
The proposed unsupervised method (SOD8s+) was evaluated against several state-of-the-art RGB-only baselines. On the new multi-spectral benchmark, the method achieved a max F-measure of 0.8458 and a Mean Absolute Error (MAE) of 0.0611, significantly outperforming RGB-only models like DCL (0.7885 maxF).
Qualitative analysis confirms that the inclusion of NIR data allows the model to accurately segment objects in pitch-black shadows or camouflaged backgrounds where RGB-only methods produce noisy or incomplete results. Furthermore, fine-tuning the model with a small subset of labeled multi-spectral data boosted the performance to a maxF of 0.9031.
Conclusion
This research demonstrates that NIR is a critical modality for robust Salient Object Detection. By using adversarial domain adaptation and CycleGAN-based data augmentation, multi-spectral models can be effectively trained despite limited labeled data. Future work may explore the integration of additional modalities, such as thermal infrared and depth, to further enhance scene understanding in extreme environments.
Summary updated