Salient Object Detection: A Benchmark

Ali Borji, Ming-Ming Cheng, Huaizu Jiang, Jia Li

Introduction

In computer vision, the concept of "visual saliency" has historically been ambiguous. Researchers have often conflated two distinct tasks: fixation prediction, which aims to predict where humans look (resulting in sparse, blob-like maps), and salient object detection (SOD), which focuses on segmenting the most important object in a scene into a dense, whole-object mask.

Because these tasks were frequently confused, models were evaluated interchangeably, leading to inconsistent claims about state-of-the-art performance. Furthermore, the field's rapid pace made it difficult to determine if newer models were genuinely improving or simply overfitting to dataset biases, such as "center bias," where subjects are perfectly centered. This paper presents a definitive benchmark to separate these tasks, standardize evaluation, and identify the architectural choices that drive genuine performance in SOD.

Methods and Evaluation Framework

Rather than proposing a new model, the authors developed a comprehensive evaluation framework to stress-test the current state of the field. They evaluated 41 existing models, including 29 specialized for SOD, 10 for fixation prediction, one for object proposal generation, and a baseline "Average Annotation Map" to quantify center bias.

To ensure a rigorous assessment, the study moved beyond standard Precision-Recall (PR) and ROC curves. The authors utilized F-measure, Mean Absolute Error (MAE), and a newer weighted F-measure ($F^w_\beta$) specifically designed to evaluate non-binary, continuous saliency maps. They also analyzed how different binarization methods, such as Adaptive Thresholding and SaliencyCut, affected final outputs and tested model robustness against "off-center" objects and "background-only" images.

Datasets

The benchmark utilized seven widely-used datasets representing varying levels of complexity and bias:

  1. MSRA10K: A massive standard featuring 10,000 annotated images.
  2. THUR15K & DUT-OMRON: Large-scale datasets for testing scalability.
  3. ECSSD: Natural images with structural complexity.
  4. JuddDB & PASCAL-S: Challenging datasets with cluttered backgrounds and minimal center bias.
  5. SED2: A unique dataset containing two salient objects per image, often located near the borders.
Examples from the THUR15K and SED2 datasets illustrating the variety of objects and the corresponding ground truth masks used for evaluation.

Key Insights

SOD vs. Fixation Prediction

The benchmark proves that fixation prediction models are poorly suited for segmenting salient objects. SOD requires distinct outputs and should be treated as a specialized figure-ground segmentation problem rather than a gaze-prediction task.

Architectural Drivers

Top-performing models almost universally utilize superpixels (regions) rather than individual pixels. Regions preserve object boundaries more effectively and allow for richer feature extraction, such as color histograms. Additionally, the study found that background priors—assuming the outer border of an image is background—are more robust than center priors.

The "Hallucination" Problem

A critical weakness identified is that most SOD models implicitly assume a salient object must exist. When presented with background-only images (e.g., a plain brick wall), models often "hallucinate" saliency by highlighting random high-contrast clutter.

The average annotation map for the SED2 dataset, illustrating the spatial distribution and inherent biases within specific benchmarks.

Key Results

  • The Leader: The DRFI (Discriminative Regional Feature Integration) model consistently outperformed competitors across nearly all datasets. By using a machine learning regression model to predict region saliency based on a 93-dimensional feature vector, it demonstrated the power of data-driven feature integration.
  • Robustness: While models like CB saw massive performance drops on off-center images, DRFI and DSR remained highly robust, proving their ability to generalize beyond simple center biases.
  • Efficiency: The study highlighted a clear trade-off between speed and accuracy. The HC model is exceptionally fast (~0.017s per image) but less accurate, while the top-tier DRFI is significantly slower (~0.697s per image).

Limitations and Future Directions

Despite rapid progress, SOD remains unsolved. Current models struggle with very small objects, objects touching the image border, and complex topologies. They also lack the semantic understanding to recognize when an image contains no salient objects. The authors suggest that the next logical step for the community is the adoption of deep learning—specifically Convolutional Neural Networks (CNNs)—to learn more powerful, semantic representations that surpass hand-crafted features.

Summary updated

Abstract

We extensively compare, qualitatively and quantitatively, 41 state-of-the-art models (29 salient object detection, 10 fixation prediction, 1 objectness, and 1 baseline) over seven challenging data sets for the purpose of benchmarking salient object detection and segmentation methods. From the results obtained so far, our evaluation shows a consistent rapid progress over the last few years in terms of both accuracy and running time. The top contenders in this benchmark significantly outperform the models identified as the best in the previous benchmark conducted three years ago. We find that the models designed specifically for salient object detection generally work better than models in closely related areas, which in turn provides a precise definition and suggests an appropriate treatment of this problem that distinguishes it from other problems. In particular, we analyze the influences of center bias and scene complexity in model performance, which, along with the hard cases for the state-of-the-art models, provide useful hints toward constructing more challenging large-scale data sets and better saliency models. Finally, we propose probable solutions for tackling several open problems, such as evaluation scores and data set bias, which also suggest future research directions in the rapidly growing field of salient object detection.
BibTeX
@article{borji2015salient,
  author = {Ali Borji and Ming-Ming Cheng and Huaizu Jiang and Jia Li},
  title = {{Salient Object Detection: A Benchmark}},
  journal = {IEEE Trans. Image Process.},
  year = {2015},
  url = {https://openreview.net/forum?id=2J-2aWL79wh},
}
Figure