Introduction
In computer vision, the concept of "visual saliency" has historically been ambiguous. Researchers have often conflated two distinct tasks: fixation prediction, which aims to predict where humans look (resulting in sparse, blob-like maps), and salient object detection (SOD), which focuses on segmenting the most important object in a scene into a dense, whole-object mask.
Because these tasks were frequently confused, models were evaluated interchangeably, leading to inconsistent claims about state-of-the-art performance. Furthermore, the field's rapid pace made it difficult to determine if newer models were genuinely improving or simply overfitting to dataset biases, such as "center bias," where subjects are perfectly centered. This paper presents a definitive benchmark to separate these tasks, standardize evaluation, and identify the architectural choices that drive genuine performance in SOD.
Methods and Evaluation Framework
Rather than proposing a new model, the authors developed a comprehensive evaluation framework to stress-test the current state of the field. They evaluated 41 existing models, including 29 specialized for SOD, 10 for fixation prediction, one for object proposal generation, and a baseline "Average Annotation Map" to quantify center bias.
To ensure a rigorous assessment, the study moved beyond standard Precision-Recall (PR) and ROC curves. The authors utilized F-measure, Mean Absolute Error (MAE), and a newer weighted F-measure ($F^w_\beta$) specifically designed to evaluate non-binary, continuous saliency maps. They also analyzed how different binarization methods, such as Adaptive Thresholding and SaliencyCut, affected final outputs and tested model robustness against "off-center" objects and "background-only" images.
Datasets
The benchmark utilized seven widely-used datasets representing varying levels of complexity and bias:
- MSRA10K: A massive standard featuring 10,000 annotated images.
- THUR15K & DUT-OMRON: Large-scale datasets for testing scalability.
- ECSSD: Natural images with structural complexity.
- JuddDB & PASCAL-S: Challenging datasets with cluttered backgrounds and minimal center bias.
- SED2: A unique dataset containing two salient objects per image, often located near the borders.
Key Insights
SOD vs. Fixation Prediction
The benchmark proves that fixation prediction models are poorly suited for segmenting salient objects. SOD requires distinct outputs and should be treated as a specialized figure-ground segmentation problem rather than a gaze-prediction task.
Architectural Drivers
Top-performing models almost universally utilize superpixels (regions) rather than individual pixels. Regions preserve object boundaries more effectively and allow for richer feature extraction, such as color histograms. Additionally, the study found that background priors—assuming the outer border of an image is background—are more robust than center priors.
The "Hallucination" Problem
A critical weakness identified is that most SOD models implicitly assume a salient object must exist. When presented with background-only images (e.g., a plain brick wall), models often "hallucinate" saliency by highlighting random high-contrast clutter.
Key Results
- The Leader: The DRFI (Discriminative Regional Feature Integration) model consistently outperformed competitors across nearly all datasets. By using a machine learning regression model to predict region saliency based on a 93-dimensional feature vector, it demonstrated the power of data-driven feature integration.
- Robustness: While models like CB saw massive performance drops on off-center images, DRFI and DSR remained highly robust, proving their ability to generalize beyond simple center biases.
- Efficiency: The study highlighted a clear trade-off between speed and accuracy. The HC model is exceptionally fast (~0.017s per image) but less accurate, while the top-tier DRFI is significantly slower (~0.697s per image).
Limitations and Future Directions
Despite rapid progress, SOD remains unsolved. Current models struggle with very small objects, objects touching the image border, and complex topologies. They also lack the semantic understanding to recognize when an image contains no salient objects. The authors suggest that the next logical step for the community is the adoption of deep learning—specifically Convolutional Neural Networks (CNNs)—to learn more powerful, semantic representations that surpass hand-crafted features.
Summary updated