Saliency Prediction with External Knowledge

Yifeng Zhang, Ming Jiang, Qi Zhao

Introduction

Saliency prediction—the task of predicting which parts of an image will catch a human's eye—has traditionally relied on deep neural networks (DNNs) trained on large eye-tracking datasets. While these models are proficient at identifying low-level features like contrast and some high-level semantics, they often lack the prior knowledge that humans naturally possess. For instance, a human observer is instinctively more likely to look at a bag a person is holding or a laptop they are using because of the semantic relationship between those objects.

Traditional models learn these relationships implicitly and purely from data, making them highly sensitive to the quality and quantity of the training set. This research addresses the gap between "data-driven" and "knowledge-driven" attention by explicitly incorporating external semantic knowledge into the prediction process.

Methodology: The GraSSNet Architecture

The authors introduce the Graph Semantic Saliency Network (GraSSNet), which mimics human attention by explicitly modeling object relationships as a graph. The architecture consists of several sophisticated stages designed to integrate external knowledge with visual features.

The GraSSNet architecture integrates regional features with a Semantic Proximity Graph derived from external knowledge bases.

Core Components

  1. Object Feature Retrieval: The model utilizes a pre-trained Faster R-CNN to detect objects and extract regional features ($h$).
  2. Semantic Proximity Network (SPN): This is the "knowledge" core. It predicts a "proximity graph" where nodes are detected objects and edges represent the strength of their semantic relationship. This network is supervised by external knowledge from MSCOCO Captions (capturing common co-occurrences) and WordNet (capturing taxonomic relationships).
  3. Spatial Graph Attention Network (sGAT): To prevent the loss of spatial information common in standard GATs, the authors developed sGAT. This module propagates semantic information across the graph while preserving the spatial characteristics of the region proposals.
  4. Saliency Map Generation: Knowledge-aware features ($m_e$) are concatenated with baseline spatial features ($m_b$) and a learned center-bias map ($b$) to produce the final prediction ($\hat{y}$).

Experimental Setup

The model was evaluated on four major public saliency benchmarks:

  • SALICON: The largest dataset, based on mouse-tracking on MSCOCO images.
  • MIT1003: 1,003 natural scenes with eye-tracking data.
  • CAT2000: 4,000 images spanning 20 different categories.
  • OSIE: 700 scenes with fixations and fine-grained object segmentations.

Results and Discussion

GraSSNet achieved state-of-the-art (SOTA) performance across all four datasets. On the SALICON dataset, it achieved a Linear Correlation Coefficient (CC) of 0.867 and a Normalized Scanpath Saliency (NSS) of 3.261, consistently outperforming other SOTA models like DINet.

Qualitative Analysis

Visually, GraSSNet excels at "spreading" saliency across related objects. While baseline models might focus only on the most visually dominant object, GraSSNet correctly identifies that if one object is salient, its semantically related neighbors are likely to be salient as well.

Qualitative comparison showing GraSSNet's ability to match ground truth (GT) more closely than the DINet baseline by correctly distributing attention across related objects.

Key Insights

  • Semantic Proximity is Key: Saliency is context-dependent. Explicitly modeling relationships allows the model to generalize better to smaller datasets where such relationships might not be well-represented in the training data.
  • Complementary Knowledge: Ablation studies revealed that using both MSCOCO and WordNet together yielded better results than using either alone, proving that co-occurrence and taxonomic knowledge are complementary.
  • Spatial Preservation: The sGAT mechanism ensures the model understands not just what is related, but where those related objects are located in the 2D image space.

Conclusion and Future Work

While GraSSNet sets a new benchmark, it remains dependent on the initial object detector's accuracy. If the detector fails to identify an object, it cannot be integrated into the semantic graph. Future research may explore dynamic knowledge acquisition or more complex "scene graphs" that model specific actions and attributes. Additionally, the framework could be extended to video saliency, where temporal relationships between objects become critical.

Summary updated

BibTeX
@inproceedings{zhang2021saliency,
  author = {Yifeng Zhang and Ming Jiang and Qi Zhao},
  title = {{Saliency Prediction with External Knowledge}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00053},
}
Figure