Saliency Prediction with External Knowledge

Yifeng Zhang, Ming Jiang, Qi Zhao

Abstract

Saliency prediction aims to identify the most visually significant regions in an image. While deep learning models have made significant strides, they often lack the structured prior knowledge humans use to navigate complex scenes. We present GraSSNet (Graph Semantic Saliency Network), which incorporates external semantic knowledge from WordNet and MSCOCO to model object relationships. By utilizing a novel Spatial Graph Attention Network (sGAT), GraSSNet achieves state-of-the-art performance on major benchmarks like SALICON and MIT1003.

Introduction

Traditional saliency models predict the "most interesting" parts of an image by relying on deep neural networks trained on large eye-tracking datasets. While effective at capturing low-level features like contrast and some high-level semantics, these models lack the structured "prior knowledge" inherent to human perception. For instance, a human understands that a person is likely to interact with a laptop or a handbag; consequently, visual attention naturally flows toward these related objects. Standard models must learn these patterns implicitly from raw data, which is often inefficient. GraSSNet addresses this gap by providing an explicit "cheat sheet" of semantic relationships to improve prediction accuracy, particularly in complex scenes.

Methods

GraSSNet introduces three primary innovations to bridge the gap between raw visual features and semantic understanding:

  1. Semantic Proximity Network (SPN): This module processes object proposals from a pre-trained Faster R-CNN to predict a "proximity graph." This graph quantifies the semantic relationship between objects. Training involves "distillation" from external knowledge bases: WordNet for taxonomic relationships (e.g., a fork and spoon are both cutlery) and MSCOCO Captions for situational co-occurrence (e.g., people are often near cars).
  2. Spatial Graph Attention Network (sGAT): Unlike standard Graph Attention Networks (GATs) that treat nodes as abstract points, sGAT uses learnable spatial filters to propagate features across the graph while preserving the 2D spatial structure of object features.
  3. Multi-branch Integration: The final saliency map is generated by fusing graph-based semantic features with a standard spatial feature map and an optional center-bias prior.
Overview of the GraSSNet architecture.

Datasets and Evaluation

The model was evaluated on four major saliency benchmarks:

  • SALICON: The largest dataset, utilizing mouse-tracking data as a proxy for eye-tracking.
  • MIT1003: 1,003 natural indoor and outdoor scenes.
  • CAT2000: 4,000 images across 20 diverse categories.
  • OSIE: 700 scenes with high-quality segmented objects.

Key Results

GraSSNet achieved state-of-the-art (SOTA) performance across all four datasets. On SALICON, it reached a Linear Correlation Coefficient (CC) of 0.867 and a Normalized Scanpath Saliency (NSS) of 3.261, outperforming previous leaders such as DINet and SAM-Res.

Qualitative comparison of GraSSNet against other state-of-the-art saliency models.

Ablation studies confirmed that removing external knowledge components led to a measurable drop in performance across various backbones (ResNet-50, ResNet-101, and VGG-19), indicating that the semantic knowledge is a critical driver of the model's success. Qualitatively, GraSSNet excels in complex images with numerous objects, such as crowded dinner tables, by better identifying the relative importance of objects compared to models relying solely on raw pixels.

Discussion and Key Insights

  • Knowledge as a Complement: External knowledge complements existing data-driven approaches. Even with a powerful ResNet-101 backbone, explicit semantic links provide a significant performance boost.
  • Human-Centric vs. Taxonomic Knowledge: MSCOCO knowledge provides a "human-centric" perspective (objects humans use), while WordNet provides "taxonomic" structure. Combining both yields optimal results.
  • Spatial Integrity: In computer vision, spatial context is paramount. The transition from standard GAT to Spatial GAT was essential for maintaining the integrity of the saliency map.

Limitations and Future Work

GraSSNet's performance is currently dependent on the accuracy of the underlying object detector (Faster R-CNN). If an object is missed or obscured, it cannot be integrated into the semantic graph. Future research may explore dynamic or task-specific relationship modeling and extending the framework to video saliency, where temporal interactions add further complexity.

Summary updated

BibTeX
@inproceedings{zhang2021saliency,
  author = {Yifeng Zhang and Ming Jiang and Qi Zhao},
  title = {{Saliency Prediction with External Knowledge}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00053},
}
Figure