Learning Synergistic Attention for Light Field Salient Object Detection

Yi Zhang, Geng Chen, Qian Chen, Yujia Sun, Yong Xia, Olivier Déforges, Wassim Hamidouche, Lu Zhang

TL;DR

This paper introduces SA-Net, a novel approach for light field salient object detection that leverages synergistic attention between all-in-focus (AiF) images and focal stacks (FSs). By employing 3D CNNs for FS feature extraction and a cascaded synergistic attention module, SA-Net effectively fuses multi-modal information, achieving state-of-the-art performance on benchmark datasets.

What Problem Does This Paper Address?

Salient object detection (SOD) aims to identify objects that capture human attention, with applications in various computer vision tasks. Light field SOD, a more recent sub-field, utilizes richer light field data beyond standard RGB images, such as focal stacks (FSs) and all-in-focus (AiF) images. While existing methods have shown promise, they often fail to fully exploit the complementary information between AiF images and FSs. Specifically, they tend to treat these modalities separately or use basic attention mechanisms, missing opportunities for deeper cross-modal interaction. Furthermore, the sequential nature of focal slices in FSs, which captures depth information, is often not adequately modeled. This paper addresses these limitations by proposing a method that synergistically fuses multi-modal features from AiF and FS data, while also effectively capturing the inter-slice information within FSs.

Methods

The proposed SA-Net consists of three main components:

  • Multi-Modal Encoder: This component uses a dual-branch architecture. A 2D CNN (ResNet50) processes the AiF images, while a 3D CNN (inflated ResNet50) is employed to extract features from the focal stacks (FSs). The 3D CNN is crucial for capturing the spatial-temporal information across the different focal slices, effectively modeling the depth variations.
  • Synergistic Attention (SA) Module: This is the core innovation. It comprises two cascaded modules:
  • Multi-Level Attention: This stage refines features from each modality individually. It uses receptive field blocks (RFBs) to enrich global context and then applies channel attention (CA) mechanisms to highlight important features.
  • Multi-Modal Attention: Inspired by mutual attention mechanisms, this stage fuses the refined features from both AiF and FS branches. It computes a similarity matrix between the two modalities and uses co-attention (CoA) to learn cross-modal interactions, enabling a synergistic effect.
  • Progressive Fusion (PF) Module: This module up-samples the high-level features from the SA modules and fuses them for the final saliency prediction. It incorporates an AiF-induced attention (AA) component to balance the features before a final deconvolutional block generates the saliency map.

The model is trained using a hybrid loss function that combines Binary Cross Entropy (BCE), Intersection over Union (IoU), and E-Measure losses.

Datasets

The paper evaluates SA-Net on three widely-used light field datasets:

  • DUT-LF: Contains 1,462 AiF images.
  • HFUT: Contains 255 AiF images.
  • LFSD: Contains 100 scenes with AiF images, FSs, multi-view images, and depth maps.

The authors follow standard training/testing splits for DUT-LF and HFUT, and use a specific split for LFSD as described in prior work.

Key Insights and Ideas

  • Synergistic Attention is Key: The paper strongly advocates for the synergistic interaction between AiF and FS features, moving beyond simple concatenation or basic attention. The proposed SA module effectively captures these complementary relationships.
  • 3D CNNs for Focal Stacks: Utilizing 3D CNNs for FS feature extraction is a novel approach in light field SOD. This allows for the effective modeling of inter-slice information and depth variations, which is crucial for accurate saliency detection.
  • High-Level Feature Fusion: The decoder focuses on fusing high-level features, which are argued to retain essential object cues like shape and location, while avoiding redundant low-level information.
  • Generalization Ability: SA-Net trained on DUT-LF and HFUT demonstrates strong performance on the unseen LFSD dataset, indicating good generalization capabilities.

Key Results

SA-Net achieves state-of-the-art performance on the evaluated datasets, outperforming 28 other methods.

  • Quantitative: The paper reports superior scores across multiple metrics, including F-Measure (Fβ), S-Measure (Sa), E-Measure (Eφ), and Mean Absolute Error (M). For instance, on DUT-LF, SA-Net achieves an Fβ of 0.920 and an Eφ of 0.032, which are the best reported scores.
  • Qualitative: Visual comparisons show that SA-Net produces saliency maps that are closest to the ground truth, with better object localization, intact structures, and fewer false positives/negatives compared to competing methods.

Limitations and Future Work

  • Computational Cost: While not explicitly stated as a limitation, the use of 3D CNNs inherently increases computational complexity compared to 2D CNNs.
  • Parameter Efficiency: The paper notes that increasing parameters by focusing on low-level features (as in F-SA) did not lead to performance improvement, suggesting a need for efficient feature fusion strategies.
  • Future Work: The paper's contributions lay the groundwork for further exploration of synergistic attention mechanisms and 3D CNNs in light field SOD. The success of the proposed SA module suggests potential for its application in other multi-modal fusion tasks.

Summary updated

Abstract

We propose a novel Synergistic Attention Network (SA-Net) to address the light field salient object detection by establishing a synergistic effect between multi-modal features with advanced attention mechanisms. Our SA-Net exploits the rich information of focal stacks via 3D convolutional neural networks, decodes the high-level features of multi-modal light field data with two cascaded synergistic attention modules, and predicts the saliency map using an effective feature fusion module in a progressive manner. Extensive experiments on three widely-used benchmark datasets show that our SA-Net outperforms 28 state-of-the-art models, sufficiently demonstrating its effectiveness and superiority. Our code is available at https://github.com/PanoAsh/SA-Net.
BibTeX
@misc{zhang2021learning,
  author = {Yi Zhang and Geng Chen and Qian Chen and Yujia Sun and Yong Xia and Olivier Déforges and Wassim Hamidouche and Lu Zhang},
  title = {{Learning Synergistic Attention for Light Field Salient Object Detection}},
  year = {2021},
  eprint = {2104.13916},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.13916},
}
Figure