Stacked Cross Refinement Network for Edge-Aware Salient Object Detection

Zhe Wu, Li Su, Qingming HuangICCV 2019 · OpenReview

Abstract

Salient object detection is a fundamental computer vision task. The majority of existing algorithms focus on aggregating multi-level features of pre-trained convolutional neural networks. Moreover, some researchers attempt to utilize edge information for auxiliary training. However, existing edge-aware models design unidirectional frameworks which only use edge features to improve the segmentation features. Motivated by the logical interrelations between binary segmentation and edge maps, we propose a novel Stacked Cross Refinement Network (SCRN) for salient object detection in this paper. Our framework aims to simultaneously refine multi-level features of salient object detection and edge detection by stacking Cross Refinement Unit (CRU). According to the logical interrelations, the CRU designs two direction-specific integration operations, and bidirectionally passes messages between the two tasks. Incorporating the refined edge-preserving features with the typical U-Net, our model detects salient objects accurately. Extensive experiments conducted on six benchmark datasets demonstrate that our method outperforms existing state-of-the-art algorithms in both accuracy and efficiency. Besides, the attribute-based performance on the SOC dataset show that the proposed model ranks first in the majority of challenging scenes.

Introduction

Salient Object Detection (SOD) aims to identify and segment the most visually significant objects in an image. A persistent challenge in SOD is the generation of clean, sharp boundaries; many existing models produce blurry heatmaps that bleed into the background. While edge information has been used to assist segmentation, previous approaches typically employed unidirectional frameworks where edge features improved the object map, but not vice versa.

The Stacked Cross Refinement Network (SCRN) addresses this by treating edge detection and object segmentation as mutually beneficial, bidirectional tasks. This approach is grounded in the logical link that an object's boundary is a subset of its total area. Consequently, spatial information from the object region can help localize edges, while precise edge information can sharpen the object's boundaries.

Methodology

The SCRN architecture utilizes a ResNet-50 backbone to extract multi-level features, which are then processed through two parallel decoders for edge and salient object detection.

Cross Refinement Unit (CRU)

The core of the SCRN is the Cross Refinement Unit (CRU), which facilitates bidirectional message passing between the two branches:

  • Segmentation to Edge ($S \to E$): Employs element-wise multiplication to ensure edge features are primarily active within the predicted object region.
  • Edge to Segmentation ($S \leftarrow E$): Uses concatenation to integrate edge information, helping to "fill in" and sharpen the object's boundaries.
Architecture of the Stacked Cross Refinement Network (SCRN) featuring the bidirectional Cross Refinement Unit (CRU).

Iterative Refinement via Stacking

To progressively clean and sharpen features, the model stacks four CRUs in sequence. This iterative process allows the network to re-evaluate features multiple times, effectively increasing the receptive field and suppressing background noise. A "Selective Set-to-Point Style" (SSPS) strategy is further employed within the CRUs to aggregate higher-level features while ignoring lower-level noise that might contain distracting background information.

Experimental Results

The SCRN was trained on the DUTS-TR dataset and evaluated across six standard benchmarks: ECSSD, PASCAL-S, DUTS-TE, HKU-IS, DUT-OMRON, and THUR15K.

Performance and Efficiency

As shown in the quantitative results, SCRN consistently outperforms ten state-of-the-art algorithms, achieving the lowest Mean Absolute Error (MAE) and highest F-measure across all tested datasets. Notably, the model maintains high efficiency, running at 52 frames per second (FPS) on a single Titan X GPU, making it significantly faster than many competing high-accuracy models.

Quantitative comparison of SCRN against state-of-the-art methods across six benchmark datasets, highlighting its superior accuracy and speed.

Robustness in Complex Scenes

Evaluations on the SOC (Salient Objects in Clutter) dataset further demonstrate the model's robustness. SCRN ranked first in the majority of challenging categories, such as low contrast and large objects, proving its capability in handling complex real-world scenarios where objects are partially occluded or blend into the background.

Conclusion

The Stacked Cross Refinement Network (SCRN) introduces a novel bidirectional refinement strategy for salient object detection. By leveraging the logical interrelations between edges and object masks through stacked CRUs, the model achieves state-of-the-art performance with high computational efficiency. This bidirectional approach provides a promising framework for other related vision tasks, such as semantic segmentation and instance boundary detection.

Summary updated

Abstract

Salient object detection is a fundamental computer vision task. The majority of existing algorithms focus on aggregating multi-level features of pre-trained convolutional neural networks. Moreover, some researchers attempt to utilize edge information for auxiliary training. However, existing edge-aware models design unidirectional frameworks which only use edge features to improve the segmentation features. Motivated by the logical interrelations between binary segmentation and edge maps, we propose a novel Stacked Cross Refinement Network (SCRN) for salient object detection in this paper. Our framework aims to simultaneously refine multi-level features of salient object detection and edge detection by stacking Cross Refinement Unit (CRU). According to the logical interrelations, the CRU designs two direction-specific integration operations, and bidirectionally passes messages between the two tasks. Incorporating the refined edge-preserving features with the typical U-Net, our model detects salient objects accurately. Extensive experiments conducted on six benchmark datasets demonstrate that our method outperforms existing state-of-the-art algorithms in both accuracy and efficiency. Besides, the attribute-based performance on the SOC dataset show that the proposed model ranks first in the majority of challenging scenes. Code can be found at https://github.com/wuzhe71/SCAN.
BibTeX
@inproceedings{wu2019stacked,
  author = {Zhe Wu and Li Su and Qingming Huang},
  title = {{Stacked Cross Refinement Network for Edge-Aware Salient Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=ZWIG0x3-Gct},
}
Figure