Backtracking Spatial Pyramid Pooling-Based Image Classifier for Weakly Supervised Top-Down Salient Object Detection

Hisham Cholakkal, Jubin Johnson, Deepu RajanIEEE TIP 2018 · OpenReview

TL;DR

This paper introduces a novel weakly supervised framework for top-down salient object detection. By leveraging binary image-level labels and a "backtracking" strategy on CNN features, it achieves performance comparable to fully supervised methods, enabling applications like semantic segmentation and object localization without pixel-level annotations.

What Problem Does This Paper Address?

Traditional "top-down" (TD) saliency models excel at identifying regions of interest relevant to a specific task, like object detection. However, these models typically require extensive pixel-level annotations of objects for training, which are costly and time-consuming to acquire. This paper addresses the challenge of achieving high-performance TD saliency detection using only weak supervision, specifically, binary labels indicating the presence or absence of an object in an image. This gap limits the scalability and applicability of TD saliency methods to scenarios where detailed annotations are unavailable.

Methods

The core of the proposed method lies in a "backtracking" strategy applied to a Spatial Pyramid Pooling (SPP)-based CNN image classifier. Here's a breakdown of the key techniques:

  • CNN Image Classifier (cSPP): A VGG-16 network, pre-trained on ImageNet, is used for feature extraction up to the relu5_3 layer. A multi-scale spatial pyramid pooling operation is applied to these features, followed by a binary linear SVM classifier trained with image-level labels. This classifier predicts the confidence of an object's presence.
  • Backtracking for TD Saliency (B-cSPP): The authors "backtrack" through the cSPP classifier to understand how each feature contributes to the final classification score. This involves estimating feature representativeness and relevance to the object class. The output is a TD saliency map that highlights regions contributing to the classifier's confidence.
  • Bottom-Up (BU) Saliency Selection: To overcome the limitations of BU saliency (which is task-agnostic and can highlight background), a novel strategy is introduced to select the "best" BU saliency map for a given task. This selection is guided by how well a BU map aligns with the object regions identified by the TD saliency.
  • Combined Saliency: The TD saliency map (B-cSPP) is combined with the selected BU saliency map. This integration aims to leverage the strengths of both approaches, reducing false positives from BU methods and improving coverage from TD methods.
  • Feature Saliency Training: A separate linear SVM classifier is trained on features extracted from regions identified by the combined saliency. This helps assign saliency to non-discriminative object regions.
  • Multi-Scale Superpixel Averaging: The saliency map is refined using multi-scale superpixel averaging to ensure smooth transitions at object boundaries and improve accuracy along them.
  • Category-Independent Saliency: By integrating category-specific saliency scores with a normalized classifier confidence, a category-independent saliency map is generated, useful for general object detection.

Datasets

The paper evaluates its framework on several standard datasets:

  • Graz-02: Used for evaluating category-specific saliency.
  • PASCAL VOC-2012: Utilized for image classification, semantic segmentation, object segmentation, object localization, and object detection tasks.
  • PASCAL VOC-2007: Used for evaluating saliency models.
  • PASCAL-S: A dataset for category-independent saliency models.
  • Object Discovery Dataset: Used for evaluating object segmentation.

Key Insights and Ideas

  • Weak Supervision is Powerful: The paper demonstrates that high-performance TD saliency can be achieved with only image-level labels, significantly reducing annotation effort.
  • Backtracking CNNs for Saliency: The novel "backtracking" approach to analyze feature contributions in a CNN classifier is a key innovation, allowing for the generation of TD saliency maps without explicit pixel-level supervision.
  • Synergy of BU and TD Saliency: Combining BU and TD saliency, with a smart selection strategy for BU maps, proves more effective than either approach alone. This highlights the complementary nature of bottom-up and top-down visual processing.
  • Feature Saliency for Completeness: Learning a dedicated feature saliency model helps to assign saliency to object parts that might not be discriminative for classification but are part of the object.
  • Superpixel Averaging for Boundary Refinement: Multi-scale superpixel averaging is crucial for producing sharp object boundaries in the final saliency maps.

Key Results

  • Comparable to Fully Supervised Methods: The proposed weakly supervised approach achieves performance comparable to, and in some cases better than, fully supervised TD saliency methods across various datasets and tasks.
  • Significant Improvements in Applications: The framework demonstrates strong performance in semantic segmentation, object segmentation, object localization, and object detection, often outperforming existing weakly supervised methods. For instance, on PASCAL VOC-2012 segmentation, it achieved a mean accuracy of 61.4%, outperforming a fully supervised method by 5%.
  • Efficiency: The method requires only one forward pass for feature extraction, making it computationally efficient compared to some other weakly supervised approaches that require multiple passes.
  • Qualitative Improvements: Visual results show that the proposed method generates accurate saliency maps that highlight objects effectively and produce sharp boundaries.

Limitations and Future Work

  • Limited Discrimination of Spatially Adjacent Instances: Similar to other weakly supervised approaches, the proposed method has limited ability to discriminate among multiple instances of an object that are spatially adjacent. This leads to lower performance in object detection compared to state-of-the-art fully supervised detectors.
  • Future Work: The authors plan to extend their framework to videos for weakly supervised salient object detection.

Summary updated

Abstract

Top-down (TD) saliency models produce a probability map that peaks at target locations specified by a task or goal such as object detection. They are usually trained in a fully supervised (FS) setting involving pixel-level annotations of objects. We propose a weakly supervised TD saliency framework using only binary labels that indicate the presence or absence of an object in an image. First, the probabilistic contribution of each image region to the confidence of a convolutional neural network-based image classifier is computed through a backtracking strategy to produce TD saliency. From a set of saliency maps of an image produced by fast bottom-up (BU) saliency approaches, we select the best saliency map suitable for the TD task. The selected BU saliency map is combined with the TD saliency map. Features having high combined saliency are used to train a linear SVM classifier to estimate feature saliency. This is integrated with combined saliency and further refined through a multi-scale superpixel averaging of saliency map. We evaluate the performance of the proposed weakly supervised TD saliency and achieve comparable performance with FS approaches. Experiments are carried out on seven challenging datasets, and quantitative results are compared with 40 closely related approaches across four different applications.
BibTeX
@article{cholakkal2018backtracking,
  author = {Hisham Cholakkal and Jubin Johnson and Deepu Rajan},
  title = {{Backtracking Spatial Pyramid Pooling-Based Image Classifier for Weakly Supervised Top-Down Salient Object Detection}},
  journal = {IEEE Trans. Image Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=jR65my2PTHWA},
}
Figure