Unified Unsupervised Salient Object Detection via Knowledge Transfer

Yao Yuan, Wutao Liu, Pan Gao, Qun Dai, Jie Qin

Abstract

Recently, unsupervised salient object detection (USOD) has gained increasing attention due to its annotation-free nature. However, current methods mainly focus on specific tasks such as RGB and RGB-D, neglecting the potential for task migration. In this paper, we propose a unified USOD framework for generic USOD tasks. Firstly, we propose a Progressive Curriculum Learning-based Saliency Distilling (PCL-SD) mechanism to extract saliency cues from a pre-trained deep network. This mechanism starts with easy samples and progressively moves towards harder ones, to avoid initial interference caused by hard samples. Afterwards, the obtained saliency cues are utilized to train a saliency detector, and we employ a Self-rectify Pseudo-label Refinement (SPR) mechanism to improve the quality of pseudo-labels. Finally, an adapter-tuning method is devised to transfer the acquired saliency knowledge, leveraging shared knowledge to attain superior transferring performance on the target tasks. Extensive experiments on five representative SOD tasks confirm the effectiveness and feasibility of our proposed method. Code and supplement materials are available at https://github.com/I2-Multimedia-Lab/A2S-v3.

Introduction

Salient Object Detection (SOD) aims to identify the most visually prominent objects in an image. While supervised methods have reached high accuracy, they rely on expensive pixel-level manual annotations. Unsupervised Salient Object Detection (USOD) circumvents this by learning from raw data, yet it faces two significant challenges: the "Hard Sample" trap and "Task Silos."

The "Hard Sample" trap occurs when complex scenes or ambiguous object boundaries confuse the model during early training, leading to error accumulation. "Task Silos" refers to the tendency of USOD research to focus on standard RGB images, often ignoring domains like Video or Remote Sensing Images (RSI) due to data scarcity. This paper introduces a unified framework to bridge these gaps through knowledge transfer and curriculum learning.

Conceptual overview of the unified USOD framework showing knowledge transfer across diverse SOD tasks.

Proposed Methodology

The proposed framework operates in two main stages: Saliency Cue Distilling and Saliency Detector Training.

1. Progressive Curriculum Learning-based Saliency Distilling (PCL-SD)

To avoid the pitfalls of hard samples, the PCL-SD mechanism implements a curriculum. Instead of training on the entire dataset simultaneously, the model initially focuses on "easy" pixels—those where the pre-trained deep network exhibits high confidence. As training progresses, the threshold is gradually lowered to include more complex "hard" samples. This phased approach prevents "pattern collapse"—where the model output becomes meaningless noise—and ensures a stable learning trajectory.

2. Self-rectify Pseudo-label Refinement (SPR)

Once saliency cues are distilled, they serve as pseudo-labels for training the final detector. To ensure high-quality labels, the SPR mechanism employs two types of rectification:

  • Posterior Rectification: The model uses its own evolving predictions to iteratively correct errors in the initial pseudo-labels.
  • Prior Rectification: A lightweight pixel refiner utilizes the original image's spatial and color information to sharpen object boundaries. Unlike traditional methods relying on computationally heavy Conditional Random Fields (CRF), this lightweight refiner ensures faster training while maintaining structural accuracy.

3. Knowledge Transfer via Adapter-Tuning

To extend the model's capabilities to specialized domains like Video SOD or RSI SOD, the authors utilize adapter-tuning. By freezing the majority of the pre-trained weights and only training small "adapter" modules, the framework can leverage shared knowledge from standard RGB images. This approach leverages the insight that different SOD tasks share common underlying logic, allowing for effective "zero-shot" transfer to new domains with minimal additional training.

The two-stage framework: Saliency Cue Distilling (Stage 1) and Saliency Detector Training (Stage 2).

Experimental Results

The framework was evaluated across five distinct SOD tasks, including RGB, RGB-D (Depth), RGB-T (Thermal), Video, and Remote Sensing.

  • State-of-the-Art Performance: The method consistently outperformed existing USOD benchmarks (e.g., TSD, A2S). On the DUTS-TE dataset, it achieved a Mean Absolute Error (MAE) of 0.047.
  • Comparison to Supervised Models: Remarkably, this unsupervised approach achieved results competitive with some weakly-supervised and fully-supervised models, demonstrating the power of effective knowledge distillation.
  • Domain Migration: The model exhibited strong "zero-shot" transfer capabilities. For instance, knowledge gained from still images was successfully applied to video sequences, with performance further enhanced through efficient adapter-tuning.

Conclusion

This research presents a robust, unified framework for unsupervised salient object detection. By addressing the "Hard Sample" trap through curriculum learning and breaking "Task Silos" via adapter-tuning, the method achieves superior performance across diverse imaging modalities. Future work will explore modality-informed training and better exploitation of temporal logic in video sequences to further refine detection accuracy.

Summary updated

Abstract

Recently, unsupervised salient object detection (USOD) has gained increasing attention due to its annotation-free nature. However, current methods mainly focus on specific tasks such as RGB and RGB-D, neglecting the potential for task migration. In this paper, we propose a unified USOD framework for generic USOD tasks. Firstly, we propose a Progressive Curriculum Learning-based Saliency Distilling (PCL-SD) mechanism to extract saliency cues from a pre-trained deep network. This mechanism starts with easy samples and progressively moves towards harder ones, to avoid initial interference caused by hard samples. Afterwards, the obtained saliency cues are utilized to train a saliency detector, and we employ a Self-rectify Pseudo-label Refinement (SPR) mechanism to improve the quality of pseudo-labels. Finally, an adapter-tuning method is devised to transfer the acquired saliency knowledge, leveraging shared knowledge to attain superior transferring performance on the target tasks. Extensive experiments on five representative SOD tasks confirm the effectiveness and feasibility of our proposed method. Code and supplement materials are available at https://github.com/I2-Multimedia-Lab/A2S-v3.
BibTeX
@inproceedings{yuan2024unified,
  author = {Yao Yuan and Wutao Liu and Pan Gao and Qun Dai and Jie Qin},
  title = {{Unified Unsupervised Salient Object Detection via Knowledge Transfer}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=Jr3hLt7IqB},
}
Figure