Multi-Scale Adversarial Feature Learning for Saliency Detection

Dandan Zhu, Lei Dai, Ye Luo, Guokai Zhang, Xuan Shao, Laurent Itti, Jianwei LuSymmetry 2018 · DOI · OpenReview

TL;DR

This paper introduces a novel "Multi-Scale Adversarial Feature Learning" (MAFL) model for image saliency detection. By leveraging Generative Adversarial Networks (GANs) with a multi-scale generator and a unique "correlation layer" in the discriminator, MAFL achieves superior performance in identifying salient regions compared to existing methods, even in complex image scenarios.

What Problem Does This Paper Address?

Visual saliency detection aims to identify the most attention-grabbing regions in an image, mimicking human visual attention. This capability is crucial for various computer vision tasks like image captioning, video summarization, and object detection. While traditional methods relied on hand-crafted features (color, intensity, texture), they struggled with complex backgrounds and semantic understanding. More recent deep learning approaches, particularly Convolutional Neural Networks (CNNs), have shown promise by automatically learning discriminative features. However, effectively capturing subtle visual contrasts and multi-scale information remains a challenge. This paper addresses the need for a more robust saliency detection method that can learn powerful, multi-scale features and better discriminate between salient and non-salient regions, especially in challenging images.

Methods

The core of this work is the Multi-Scale Adversarial Feature Learning (MAFL) model, built upon the Generative Adversarial Network (GAN) framework. MAFL consists of two main components:

  • Multi-Scale Generator Network (G-network): Unlike standard GANs that generate natural images, the G-network in MAFL is designed to generate synthetic saliency maps. It takes natural images as input and is architected to extract features at multiple scales. This is achieved by incorporating four pooling layers and, crucially, adding three extra convolutional layers after each of the first four max-pooling layers. These extra layers help capture subtle visual contrasts at different feature levels. The feature maps from these multiple scales are then fused to produce the final synthetic saliency map.
  • Discriminator Network (D-network): The D-network's role is to distinguish between real ground-truth saliency maps and the synthetic ones generated by the G-network. The key innovation here is the introduction of a novel "correlation layer." This layer directly compares patches from the ground-truth saliency map with corresponding patches in the synthetic saliency map, calculating their similarity at a pixel-level. This enhanced discrimination capability is vital for training the G-network to produce more realistic saliency maps.

Datasets

The authors evaluated their MAFL model on three public datasets:

  • THUR15K: Contains 6232 images, notable for including images without salient targets.
  • MSRA-A: Comprises 1000 images with corresponding manual annotations.
  • PASCAL-S: Derived from the PASCAL VOC 2010 segmentation challenge, featuring 850 images with multiple targets and complex backgrounds.

Key Insights and Ideas

  • GANs for Saliency Synthesis: The paper demonstrates that GANs, typically used for image generation, can be effectively repurposed for generating saliency maps, a novel application. This adversarial learning paradigm proves powerful for learning discriminative features for saliency detection.
  • Multi-Scale Feature Extraction is Crucial: The MAFL model's success hinges on its ability to extract features at multiple scales. This is essential because salient objects can appear at various sizes and contexts within an image.
  • The Power of the Correlation Layer: The introduction of the correlation layer in the discriminator significantly enhances its ability to discern subtle differences between real and generated saliency maps, leading to more accurate feature learning.
  • Improved Performance in Complex Scenarios: The MAFL model shows remarkable performance on images with cluttered backgrounds, low contrast between target and background, and multiple salient objects, areas where previous methods often faltered.

Key Results

  • Quantitative Performance: The MAFL model achieved superior results across all three benchmark datasets when compared to ten state-of-the-art saliency detection methods.
  • On the THUR15K dataset, MAFL achieved a precision of 94.46% and a recall of 98.59%.
  • The model consistently produced the highest F-measure values, indicating a strong balance between precision and recall.
  • MAFL also achieved the lowest Mean Absolute Error (MAE) on all datasets, demonstrating the accuracy of its generated saliency maps. Notably, it reduced the MAE by 3.03% on the PASCAL-S dataset compared to the best previous method.
  • Qualitative Performance: Visualizations showed that MAFL could uniformly highlight entire salient objects while preserving their structural details, even in challenging images.
  • Efficiency: The MAFL method also demonstrated the shortest execution time among the compared methods, suggesting a good trade-off between performance and computational cost.

Limitations and Future Work

  • Limitations: The paper does not explicitly detail limitations beyond the inherent challenges of saliency detection itself. However, the reliance on GANs implies potential issues with training stability and hyperparameter tuning, which are common in GAN research.
  • Future Work: The authors suggest that their MAFL model can be further improved by exploring different network architectures and potentially incorporating more sophisticated feature fusion techniques. The success of the multi-scale approach also opens avenues for exploring even finer-grained multi-scale feature extraction.

Summary updated

Abstract

Previous saliency detection methods usually focused on extracting powerful discriminative features to describe images with a complex background. Recently, the generative adversarial network (GAN) has shown a great ability in feature learning for synthesizing high quality natural images. Since the GAN shows a superior feature learning ability, we present a new multi-scale adversarial feature learning (MAFL) model for image saliency detection. In particular, we build this model, which is composed of two convolutional neural network (CNN) modules: the multi-scale G-network takes natural images as inputs and generates the corresponding synthetic saliency map, and we design a novel layer in the D-network, namely a correlation layer, which is used to determine whether one image is a synthetic saliency map or ground-truth saliency map. Quantitative and qualitative comparisons on several public datasets show the superiority of our approach.
BibTeX
@article{zhu2018multi,
  author = {Dandan Zhu and Lei Dai and Ye Luo and Guokai Zhang and Xuan Shao and Laurent Itti and Jianwei Lu},
  title = {{Multi-Scale Adversarial Feature Learning for Saliency Detection}},
  journal = {Symmetry},
  year = {2018},
  doi = {10.3390/sym10100457},
}
Figure