How is Visual Attention Influenced by Text Guidance? Database and Model

Yinan Sun, Xiongkuo Min, Huiyu Duan, Guangtao Zhai

Abstract

Traditional visual saliency models primarily focus on "bottom-up" features like color and contrast. However, real-world images are often accompanied by text that provides "top-down" guidance, significantly altering human gaze. This paper introduces SJTU-TIS, a comprehensive database of 1,200 text-image pairs with eye-tracking data, and TGSal, a multimodal model that integrates text and image features to predict visual saliency with state-of-the-art accuracy.

Introduction: The Impact of Text Guidance

In computer vision, "visual saliency" identifies image regions that naturally attract human attention. While bottom-up attention (driven by low-level features) is well-studied, the influence of top-down guidance from text—such as captions or headlines—remains under-explored. Text can force a viewer to ignore a visually dominant object in favor of a subtle detail mentioned in the description. Despite its importance, there has been a significant gap in high-quality datasets and robust models that can predict how text influences gaze on the same image.

Visual attention heatmaps demonstrating how text guidance can shift focus between different subjects in an image.

The SJTU-TIS Database

To bridge the gap in available data, the authors developed the SJTU-TIS database. It contains 600 images paired with 1,200 text descriptions, categorized to test various guidance levels:

  • Type 1: General scenario descriptions (e.g., "Birds flock over a dock").
  • Type 2: Descriptions of naturally salient objects.
  • Type 3: Descriptions of non-salient (background) objects.
  • Type 4: Mixed descriptions containing both salient and non-salient information.

Eye-tracking data from 60 participants provides a ground-truth map of how these different text types steer visual attention, confirming that human gaze is highly plastic and responsive to linguistic cues.

The TGSal Model Architecture

The proposed Text-Guided Saliency (TGSal) model employs a dual-encoder architecture to fuse visual and linguistic information:

  1. Image Encoder: Uses a ResNet-50 backbone to extract multi-scale visual features.
  2. Text Encoder: Leverages a pre-trained CLIP model to capture both global and local semantic features from the text.
  3. Fusion Mechanism: The model introduces Global and Local Text Feature Fusion (GTFF and LTFF) modules. These use cross-attention to inject text-based cues into the image features during the decoding process, allowing the model to dynamically adjust its focus based on specific words.

Key Results and Performance

TGSal demonstrates superior performance over ten state-of-the-art saliency models across standard metrics (including AUC-J, sAUC, CC, and NSS). Notably, the model excels in predicting attention shifts toward non-salient objects—the most challenging scenario for traditional models where text points to background elements the visual system might otherwise ignore.

Comparison between ground-truth human fixation maps and the saliency maps predicted by the TGSal model.

The model remains robust even without text input, performing as a top-tier "pure" image saliency predictor. Ablation studies confirmed that local text features (specific words) are more critical for accuracy than global summaries, as they allow the model to pinpoint specific objects mentioned in the text.

Conclusion

The SJTU-TIS database and TGSal model represent a significant advancement in understanding top-down visual attention. By effectively integrating text guidance, TGSal more accurately reflects how humans process information in multimodal contexts. Future work will explore dynamic guidance in video sequences and more complex linguistic structures, such as metaphors or emotional language.

Summary updated

Abstract

The analysis and prediction of visual attention have long been crucial tasks in the fields of computer vision and image processing. In practical applications, images are generally accompanied by various text descriptions, however, few studies have explored the influence of text descriptions on visual attention, let alone developed visual saliency prediction models considering text guidance. In this paper, we conduct a comprehensive study on text-guided image saliency (TIS) from both subjective and objective perspectives. Specifically, we construct a TIS database named SJTU-TIS, which includes 1200 text-image pairs and the corresponding collected eye-tracking data. Based on the established SJTU-TIS database, we analyze the influence of various text descriptions on visual attention. Then, to facilitate the development of saliency prediction models considering text influence, we construct a benchmark for the established SJTU-TIS database using state-of-the-art saliency models. Finally, considering the effect of text descriptions on visual attention, while most existing saliency models ignore this impact, we further propose a text-guided saliency (TGSal) prediction model, which extracts and integrates both image features and text features to predict the image saliency under various text-description conditions. Our proposed model significantly outperforms the state-of-the-art saliency models on both the SJTU-TIS database and the pure image saliency databases in terms of various evaluation metrics. The SJTU-TIS database and the code of the proposed TGSal model will be released at: https://github.com/IntMeGroup/TGSal.
BibTeX
@article{sun2024how,
  author = {Yinan Sun and Xiongkuo Min and Huiyu Duan and Guangtao Zhai},
  title = {{How is Visual Attention Influenced by Text Guidance? Database and Model}},
  journal = {IEEE Trans. Image Process.},
  year = {2024},
  doi = {10.1109/tip.2024.3461956},
}
Figure