Abstract
Traditional visual saliency models primarily focus on "bottom-up" features like color and contrast. However, real-world images are often accompanied by text that provides "top-down" guidance, significantly altering human gaze. This paper introduces SJTU-TIS, a comprehensive database of 1,200 text-image pairs with eye-tracking data, and TGSal, a multimodal model that integrates text and image features to predict visual saliency with state-of-the-art accuracy.
Introduction: The Impact of Text Guidance
In computer vision, "visual saliency" identifies image regions that naturally attract human attention. While bottom-up attention (driven by low-level features) is well-studied, the influence of top-down guidance from text—such as captions or headlines—remains under-explored. Text can force a viewer to ignore a visually dominant object in favor of a subtle detail mentioned in the description. Despite its importance, there has been a significant gap in high-quality datasets and robust models that can predict how text influences gaze on the same image.
The SJTU-TIS Database
To bridge the gap in available data, the authors developed the SJTU-TIS database. It contains 600 images paired with 1,200 text descriptions, categorized to test various guidance levels:
- Type 1: General scenario descriptions (e.g., "Birds flock over a dock").
- Type 2: Descriptions of naturally salient objects.
- Type 3: Descriptions of non-salient (background) objects.
- Type 4: Mixed descriptions containing both salient and non-salient information.
Eye-tracking data from 60 participants provides a ground-truth map of how these different text types steer visual attention, confirming that human gaze is highly plastic and responsive to linguistic cues.
The TGSal Model Architecture
The proposed Text-Guided Saliency (TGSal) model employs a dual-encoder architecture to fuse visual and linguistic information:
- Image Encoder: Uses a ResNet-50 backbone to extract multi-scale visual features.
- Text Encoder: Leverages a pre-trained CLIP model to capture both global and local semantic features from the text.
- Fusion Mechanism: The model introduces Global and Local Text Feature Fusion (GTFF and LTFF) modules. These use cross-attention to inject text-based cues into the image features during the decoding process, allowing the model to dynamically adjust its focus based on specific words.
Key Results and Performance
TGSal demonstrates superior performance over ten state-of-the-art saliency models across standard metrics (including AUC-J, sAUC, CC, and NSS). Notably, the model excels in predicting attention shifts toward non-salient objects—the most challenging scenario for traditional models where text points to background elements the visual system might otherwise ignore.
The model remains robust even without text input, performing as a top-tier "pure" image saliency predictor. Ablation studies confirmed that local text features (specific words) are more critical for accuracy than global summaries, as they allow the model to pinpoint specific objects mentioned in the text.
Conclusion
The SJTU-TIS database and TGSal model represent a significant advancement in understanding top-down visual attention. By effectively integrating text guidance, TGSal more accurately reflects how humans process information in multimodal contexts. Future work will explore dynamic guidance in video sequences and more complex linguistic structures, such as metaphors or emotional language.
Summary updated