Adaptive Graph Convolution Module for Salient Object Detection

Yongwoo Lee, Minhyeok Lee, Suhwan Cho, Sangyoun Lee

Abstract

Salient object detection (SOD) involves identifying and segmenting the most visually prominent objects in an image. While multi-scale feature fusion is a common approach, it often fails to account for image structures and long-range relationships between distant pixels. We propose an adaptive graph convolution module (AGCM) to address these limitations. By extracting prototype features through a learnable region generation layer and refining them via graph-based information propagation, our method achieves significant performance gains. Experimental results demonstrate that AGCM improves SOD performance both quantitatively and qualitatively.

Introduction

Salient Object Detection (SOD) aims to automatically highlight image regions that naturally attract human attention. Traditional deep learning models typically rely on multi-scale feature fusion, processing images at various resolutions to capture context. However, these methods often overlook the underlying structural relationships within an image. Standard convolutional neural networks (CNNs) are proficient at identifying local patterns but struggle with long-range dependencies, especially in complex scenes with multiple similar objects or intricate backgrounds. To overcome this, we introduce the Adaptive Graph Convolution Module (AGCM), which treats semantic regions as nodes in a graph to capture global context more effectively.

Methodology

The AGCM is a "plug-and-play" component designed for integration into standard encoder-decoder architectures, such as VGG16. The module operates in two primary stages:

  1. Prototype Feature Generation: Instead of processing individual pixels, the model employs a differentiable, learnable region generation layer to group features into "prototypes." These prototypes represent semantically and spatially related regions, allowing the network to learn optimal groupings during training.
  2. Prototype Feature Refinement: These prototypes are treated as nodes in a graph. Using EdgeConv, the model propagates information between nodes to calculate correspondence scores (an adjacency matrix). This enables the network to understand relationships between distant regions, independent of their spatial location.
Overall architecture with AGCM in skip connections.

By shifting from grid-based to graph-based processing, the model gains a higher-level structural prior while reducing computational complexity. The AGCM is also designed to be permutation invariant, ensuring that the recognition of relationships between objects is not dependent on their specific order in the feature map. Strategic placement of the AGCM in the two highest-level skip connections was found to provide the optimal balance between performance and efficiency.

Experimental Setup

The proposed model was evaluated on several standard benchmark datasets:

  • DUTS: The largest SOD dataset, utilizing DUTS-TR (10,553 images) for training and DUTS-TE (5,019 images) for testing.
  • ECSSD: Contains 1,000 images featuring semantically meaningful objects.
  • DUT-OMRON: Includes 5,168 images characterized by complex backgrounds, presenting a significant challenge for SOD models.

Results and Discussion

The AGCM-based model was compared against six state-of-the-art (SOTA) methods, including U2Net and BASNet.

Quantitative Performance

On the DUTS-TE dataset, the proposed method achieved an F-measure ($F_\beta$) of 0.826, notably outperforming the previous best-performing model, U2Net (0.792). Ablation studies further confirmed that even a single AGCM module significantly boosts performance over the baseline architecture.

Qualitative Analysis

Qualitatively, the model excels in handling complex scenes with multiple distinct salient objects, such as groups of animals or people. While competing models often produce blurry masks or miss objects entirely, the AGCM produces sharp, complete segmentations by leveraging its global understanding of the scene.

Qualitative comparison of SOD results.

Conclusion and Future Work

The Adaptive Graph Convolution Module represents a significant step forward in capturing long-range dependencies for SOD. While the results are promising, future work could explore the use of more modern backbones, such as Transformers or ResNeXt, to further enhance performance. Additionally, while the current implementation optimizes for efficiency by targeting high-level skip connections, further research into reducing the computational overhead of graph convolutions and improving performance in extremely low-contrast environments remains valuable.

Summary updated

Abstract

Salient object detection (SOD) is a task that involves identifying and segmenting the most visually prominent object in an image. Existing solutions can accomplish this use a multi-scale feature fusion mechanism to detect the global context of an image. However, as there is no consideration of the structures in the image nor the relations between distant pixels, conventional methods cannot deal with complex scenes effectively. In this paper, we propose an adaptive graph convolution module (AGCM) to overcome these limitations. Prototype features are initially extracted from the input image using a learnable region generation layer that spatially groups features in the image. The prototype features are then refined by propagating information between them based on a graph architecture, where each feature is regarded as a node. Experimental results show that the proposed AGCM dramatically improves the SOD performance both quantitatively and quantitatively.
BibTeX
@inproceedings{lee2023adaptive,
  author = {Yongwoo Lee and Minhyeok Lee and Suhwan Cho and Sangyoun Lee},
  title = {{Adaptive Graph Convolution Module for Salient Object Detection}},
  booktitle = {ICIP},
  year = {2023},
  doi = {10.1109/icip49359.2023.10222238},
}
Figure