TL;DR
This paper introduces a novel weakly supervised framework for top-down salient object detection. By leveraging binary image-level labels and a "backtracking" strategy on CNN features, it achieves performance comparable to fully supervised methods, enabling applications like semantic segmentation and object localization without pixel-level annotations.
What Problem Does This Paper Address?
Traditional "top-down" (TD) saliency models excel at identifying regions of interest relevant to a specific task, like object detection. However, these models typically require extensive pixel-level annotations of objects for training, which are costly and time-consuming to acquire. This paper addresses the challenge of achieving high-performance TD saliency detection using only weak supervision, specifically, binary labels indicating the presence or absence of an object in an image. This gap limits the scalability and applicability of TD saliency methods to scenarios where detailed annotations are unavailable.
Methods
The core of the proposed method lies in a "backtracking" strategy applied to a Spatial Pyramid Pooling (SPP)-based CNN image classifier. Here's a breakdown of the key techniques:
- CNN Image Classifier (cSPP): A VGG-16 network, pre-trained on ImageNet, is used for feature extraction up to the
relu5_3layer. A multi-scale spatial pyramid pooling operation is applied to these features, followed by a binary linear SVM classifier trained with image-level labels. This classifier predicts the confidence of an object's presence. - Backtracking for TD Saliency (B-cSPP): The authors "backtrack" through the cSPP classifier to understand how each feature contributes to the final classification score. This involves estimating feature representativeness and relevance to the object class. The output is a TD saliency map that highlights regions contributing to the classifier's confidence.
- Bottom-Up (BU) Saliency Selection: To overcome the limitations of BU saliency (which is task-agnostic and can highlight background), a novel strategy is introduced to select the "best" BU saliency map for a given task. This selection is guided by how well a BU map aligns with the object regions identified by the TD saliency.
- Combined Saliency: The TD saliency map (B-cSPP) is combined with the selected BU saliency map. This integration aims to leverage the strengths of both approaches, reducing false positives from BU methods and improving coverage from TD methods.
- Feature Saliency Training: A separate linear SVM classifier is trained on features extracted from regions identified by the combined saliency. This helps assign saliency to non-discriminative object regions.
- Multi-Scale Superpixel Averaging: The saliency map is refined using multi-scale superpixel averaging to ensure smooth transitions at object boundaries and improve accuracy along them.
- Category-Independent Saliency: By integrating category-specific saliency scores with a normalized classifier confidence, a category-independent saliency map is generated, useful for general object detection.
Datasets
The paper evaluates its framework on several standard datasets:
- Graz-02: Used for evaluating category-specific saliency.
- PASCAL VOC-2012: Utilized for image classification, semantic segmentation, object segmentation, object localization, and object detection tasks.
- PASCAL VOC-2007: Used for evaluating saliency models.
- PASCAL-S: A dataset for category-independent saliency models.
- Object Discovery Dataset: Used for evaluating object segmentation.
Key Insights and Ideas
- Weak Supervision is Powerful: The paper demonstrates that high-performance TD saliency can be achieved with only image-level labels, significantly reducing annotation effort.
- Backtracking CNNs for Saliency: The novel "backtracking" approach to analyze feature contributions in a CNN classifier is a key innovation, allowing for the generation of TD saliency maps without explicit pixel-level supervision.
- Synergy of BU and TD Saliency: Combining BU and TD saliency, with a smart selection strategy for BU maps, proves more effective than either approach alone. This highlights the complementary nature of bottom-up and top-down visual processing.
- Feature Saliency for Completeness: Learning a dedicated feature saliency model helps to assign saliency to object parts that might not be discriminative for classification but are part of the object.
- Superpixel Averaging for Boundary Refinement: Multi-scale superpixel averaging is crucial for producing sharp object boundaries in the final saliency maps.
Key Results
- Comparable to Fully Supervised Methods: The proposed weakly supervised approach achieves performance comparable to, and in some cases better than, fully supervised TD saliency methods across various datasets and tasks.
- Significant Improvements in Applications: The framework demonstrates strong performance in semantic segmentation, object segmentation, object localization, and object detection, often outperforming existing weakly supervised methods. For instance, on PASCAL VOC-2012 segmentation, it achieved a mean accuracy of 61.4%, outperforming a fully supervised method by 5%.
- Efficiency: The method requires only one forward pass for feature extraction, making it computationally efficient compared to some other weakly supervised approaches that require multiple passes.
- Qualitative Improvements: Visual results show that the proposed method generates accurate saliency maps that highlight objects effectively and produce sharp boundaries.
Limitations and Future Work
- Limited Discrimination of Spatially Adjacent Instances: Similar to other weakly supervised approaches, the proposed method has limited ability to discriminate among multiple instances of an object that are spatially adjacent. This leads to lower performance in object detection compared to state-of-the-art fully supervised detectors.
- Future Work: The authors plan to extend their framework to videos for weakly supervised salient object detection.
Summary updated