Introduction
Saliency prediction—the task of predicting which parts of an image will catch a human's eye—has traditionally relied on deep neural networks (DNNs) trained on large eye-tracking datasets. While these models are proficient at identifying low-level features like contrast and some high-level semantics, they often lack the prior knowledge that humans naturally possess. For instance, a human observer is instinctively more likely to look at a bag a person is holding or a laptop they are using because of the semantic relationship between those objects.
Traditional models learn these relationships implicitly and purely from data, making them highly sensitive to the quality and quantity of the training set. This research addresses the gap between "data-driven" and "knowledge-driven" attention by explicitly incorporating external semantic knowledge into the prediction process.
Methodology: The GraSSNet Architecture
The authors introduce the Graph Semantic Saliency Network (GraSSNet), which mimics human attention by explicitly modeling object relationships as a graph. The architecture consists of several sophisticated stages designed to integrate external knowledge with visual features.
Core Components
- Object Feature Retrieval: The model utilizes a pre-trained Faster R-CNN to detect objects and extract regional features ($h$).
- Semantic Proximity Network (SPN): This is the "knowledge" core. It predicts a "proximity graph" where nodes are detected objects and edges represent the strength of their semantic relationship. This network is supervised by external knowledge from MSCOCO Captions (capturing common co-occurrences) and WordNet (capturing taxonomic relationships).
- Spatial Graph Attention Network (sGAT): To prevent the loss of spatial information common in standard GATs, the authors developed sGAT. This module propagates semantic information across the graph while preserving the spatial characteristics of the region proposals.
- Saliency Map Generation: Knowledge-aware features ($m_e$) are concatenated with baseline spatial features ($m_b$) and a learned center-bias map ($b$) to produce the final prediction ($\hat{y}$).
Experimental Setup
The model was evaluated on four major public saliency benchmarks:
- SALICON: The largest dataset, based on mouse-tracking on MSCOCO images.
- MIT1003: 1,003 natural scenes with eye-tracking data.
- CAT2000: 4,000 images spanning 20 different categories.
- OSIE: 700 scenes with fixations and fine-grained object segmentations.
Results and Discussion
GraSSNet achieved state-of-the-art (SOTA) performance across all four datasets. On the SALICON dataset, it achieved a Linear Correlation Coefficient (CC) of 0.867 and a Normalized Scanpath Saliency (NSS) of 3.261, consistently outperforming other SOTA models like DINet.
Qualitative Analysis
Visually, GraSSNet excels at "spreading" saliency across related objects. While baseline models might focus only on the most visually dominant object, GraSSNet correctly identifies that if one object is salient, its semantically related neighbors are likely to be salient as well.
Key Insights
- Semantic Proximity is Key: Saliency is context-dependent. Explicitly modeling relationships allows the model to generalize better to smaller datasets where such relationships might not be well-represented in the training data.
- Complementary Knowledge: Ablation studies revealed that using both MSCOCO and WordNet together yielded better results than using either alone, proving that co-occurrence and taxonomic knowledge are complementary.
- Spatial Preservation: The sGAT mechanism ensures the model understands not just what is related, but where those related objects are located in the 2D image space.
Conclusion and Future Work
While GraSSNet sets a new benchmark, it remains dependent on the initial object detector's accuracy. If the detector fails to identify an object, it cannot be integrated into the semantic graph. Future research may explore dynamic knowledge acquisition or more complex "scene graphs" that model specific actions and attributes. Additionally, the framework could be extended to video saliency, where temporal relationships between objects become critical.
Summary updated