Saliency Detection via Global Context Enhanced Feature Fusion and Edge Weighted Loss

Chaewon Park, Minhyeok Lee, MyeongAh Cho, Sangyoun Lee

TL;DR

Researchers from Yonsei University have developed a more efficient way to perform Salient Object Detection (SOD) by rethinking how features are fused and how boundaries are learned. Their approach, featuring a Context Fusion Decoder Network (CFDN) and a Near Edge Weighted Loss (NEWLoss), achieves state-of-the-art results without the computational overhead of traditional edge-reconstruction branches.

What Problem Does This Paper Address?

Salient Object Detection (SOD) aims to identify and segment the most visually prominent objects in an image—much like how human attention works. While U-Net-based architectures are the standard for this task, they suffer from two major drawbacks:

  1. The Information Gap: U-Net uses "skip connections" to pass spatial details from the encoder to the decoder. However, the encoder contains details for all objects in a scene, while the decoder only cares about the salient one. This "indiscriminate integration" often introduces noise, where details of non-salient background objects (like reflections in water) blur the final prediction.
  2. The Cost of Sharp Edges: To get crisp boundaries, most models add extra sub-networks specifically for edge detection. While effective, these additional branches increase the model's size and slow down inference time.

Methods

The authors propose two novel components to solve these issues:

1. Context Fusion Decoder Network (CFDN)

Instead of blindly merging encoder and decoder features, the CFDN uses global context as a filter.

  • Context Module (CM): This module extracts a "Global Salient Context Feature" (GSCF) from the deepest layer of the encoder using global average pooling and an attention mechanism.
  • Feature Fusion Module (FFM): This module acts as the gatekeeper. It takes the noisy spatial features from the encoder and uses the GSCF as a guide to suppress non-salient information. It employs parallel convolutional layers with different dilation rates (inspired by ASPP) to capture multi-scale information effectively.

2. Near Edge Weighted Loss (NEWLoss)

To get sharp boundaries without adding extra layers to the model, the authors moved the "edge logic" into the loss function.

  • Near-Edge Mask (NEM): Using distance transforms on the ground truth, they calculate an "inner distance" and "outer distance" from the object boundary.
  • The Loss: They combine these to create a weight map (NEM) that highlights the pixels right on the edge of the object. During training, the Binary Cross Entropy (BCE) loss is multiplied by this weight map, forcing the model to pay much closer attention to—and penalize errors on—ambiguous boundaries.

Datasets

The model was trained on the DUTS-TR dataset (10,553 images) and evaluated on four standard benchmarks:

  • ECSSD: 1,000 images of semantically meaningful objects.
  • PASCAL-S: 850 images from the PASCAL VOC dataset.
  • DUT-OMRON: 5,168 images with complex backgrounds and multiple objects.
  • DUTS-TE: The test set of the DUTS dataset (5,019 images).

Key Insights and Ideas

  • Context as a Denoiser: The "aha moment" here is treating global context not just as extra information, but as a masking signal for the noisy skip connections. By knowing what the salient object is globally, the model can ignore the high-resolution details of background objects provided by the encoder.
  • Loss vs. Architecture: The paper demonstrates that you don't always need a more complex model to solve a specific problem (like blurry edges). By designing a smarter loss function (NEWLoss), you can achieve the same "supervision" effect on boundaries during training without any extra cost during inference.
  • Addressing the Information Gap: The authors highlight that the gap between high-level semantic features (decoder) and low-level spatial features (encoder) is a primary source of error in SOD, and bridging it requires active guidance rather than simple concatenation.

Key Results

  • Quantitative: The proposed method outperformed or matched state-of-the-art models (like EGNet, MINet, and AFNet) across all four benchmarks. Notably, it achieved the highest Structure Measure ($S_\alpha$) scores, indicating superior global structure in the generated masks.
  • Qualitative: Visual comparisons show the model is significantly better at ignoring background noise (like reflections) and segmenting thin, complex structures (like the legs of a chair or a bird's beak) compared to previous methods.
  • Efficiency: Because the edge-learning is handled by the loss function, the model remains streamlined and faster than models requiring dedicated edge-reconstruction branches.

Limitations and Future Work

The paper does not explicitly detail specific failure cases or limitations. However, based on the methodology:

  • Distance Transform Sensitivity: The NEWLoss relies heavily on accurate ground truth masks to calculate distance transforms; noisy ground truth could potentially lead to biased edge weighting.
  • Backbone Dependency: The model uses EfficientNet-b5 as a backbone. While powerful, future work could explore how these modules perform with even lighter backbones for mobile or real-time applications.
  • Complex Scenes: While it handles multiple objects well, extremely cluttered scenes where saliency is subjective remain a general challenge for the field.

Summary updated

Abstract

UNet-based methods have shown outstanding performance in salient object detection (SOD), but are problematic in two aspects. 1) Indiscriminately integrating the encoder feature, which contains spatial information for multiple objects, and the decoder feature, which contains global information of the salient object, is likely to convey unnecessary details of non-salient objects to the decoder, hindering saliency detection. 2) To deal with ambiguous object boundaries and generate accurate saliency maps, the model needs additional branches, such as edge reconstructions, which leads to increasing computational cost. To address the problems, we propose a context fusion decoder network (CFDN) and near edge weighted loss (NEWLoss) function. The CFDN creates an accurate saliency map by integrating global context information and thus suppressing the influence of the unnecessary spatial information. NEWLoss accelerates learning of obscure boundaries without additional modules by generating weight maps on object boundaries. Our method is evaluated on four benchmarks and achieves state-of-the-art performance. We prove the effectiveness of the proposed method through comparative experiments.
BibTeX
@inproceedings{park2022saliency,
  author = {Chaewon Park and Minhyeok Lee and MyeongAh Cho and Sangyoun Lee},
  title = {{Saliency Detection via Global Context Enhanced Feature Fusion and Edge Weighted Loss}},
  booktitle = {ICIP},
  year = {2022},
  doi = {10.1109/icip46576.2022.9897797},
}

Loading related content...

Figure