The Roles of Contextual Semantic Relevance Metrics in Human Visual Processing

Kun Sun, Wang RongarXiv 2024 · DOI · arXiv

TL;DR

Researchers have developed new metrics to predict human eye movements by combining state-of-the-art vision and language models (like CLIP and SBERT). The study reveals that a "combined" metric—integrating both visual context and linguistic meaning—is the most powerful predictor of human visual attention, proving that our brains don't just process pixels, but complex semantic relationships.

What Problem Does This Paper Address?

When you look at a kitchen scene, why does your eye gravitate toward the knife on the counter rather than a random tile on the wall? Historically, researchers explained this using "visual saliency"—low-level features like high contrast or bright colors. Later, they added "semantic relevance," the idea that we look at things that "make sense" in a scene.

However, older models of semantic relevance were often simplistic. They might look at an object in isolation or use basic word-matching. They failed to capture the nuanced context: how a "baby" relates to a "lemon" specifically within the context of a "restaurant" (as seen in the paper’s examples). This paper addresses the gap by using modern Deep Learning to compute "contextual semantic relevance" and asks: Do vision-based models or language-based models better predict human fixations? Or do we need both?

Methods

The authors moved beyond simple human ratings, instead leveraging "foundation models" to compute three types of metrics:

  1. Vision-Based Metrics: Using CLIP (Contrastive Language-Image Pre-training), they generated embeddings for the whole image and individual objects. They calculated the cosine similarity between an object and its entire scene, as well as between an object and its immediate neighbors.
  2. Language-Based Metrics: Using Sentence Transformers (SBERT) and FastText, they compared the name of a target object (e.g., "baby") against the image's caption (e.g., "a baby looking at a lemon in a restaurant") and the names of surrounding objects.
  3. The Combined Metric (total_vissem_sim): A linear integration of both vision and language scores.

To see if these metrics actually predicted human behavior, they used Generalized Additive Mixed Models (GAMMs). This statistical approach is more flexible than standard regression, allowing the researchers to map complex, non-linear relationships between "meaning" and "eye fixations."

Datasets

The study utilized the "Human Attention in Image Captioning" dataset.

  • Scale: 1,000 diverse images.
  • Data Points: Includes raw eye-tracking data (fixation duration and number) from humans describing the images.
  • Uniqueness: Unlike datasets that use simple search tasks, this one involves "naturalistic visual processing"—humans actually trying to comprehend and describe a scene, which triggers more complex cognitive processing.

Key Insights and Ideas

  • The Multi-Modal Brain: The most significant finding is that the combined metric outperformed everything else. This suggests that human visual processing isn't just "vision" or just "knowledge"; it’s a synergistic interaction. We use language-like categories to organize what we see, and visual context to ground what we know.
  • Linear Vision vs. Complex Language: Vision-based metrics showed relatively straightforward, predictable impacts on fixations. However, language-based metrics exhibited "intricate, non-linear patterns." For example, an object might be ignored if it's too irrelevant, but also if it's too predictable, creating a "U-shaped" curve of attention.
  • Top-Down vs. Bottom-Up: The authors frame vision-based metrics as proxies for bottom-up processing (sensory input) and language-based metrics as top-down processing (prior knowledge and expectations). Their results suggest top-down processing may dominate during complex tasks like image captioning.

Key Results

The researchers used $\Delta$AIC (Akaike Information Criterion) to compare models—where a more negative number indicates a better fit for the data.

  • The Winner: The combined metric (total_vissem_sim) achieved the best scores ($\Delta$AIC of -573.04 for duration and -559.2 for fixation count).
  • Vision vs. Language: Language-based metrics generally outperformed pure vision-based metrics in predicting fixations, highlighting the importance of "meaning" over "appearance" in visual comprehension.
  • Control Variables: Traditional factors like "object proportion" (size) and "saliency" remained significant, but the new semantic metrics provided substantial predictive power above and beyond these basic features.

Limitations and Future Work

The authors noted that the choice of the "semantic database" matters immensely. For instance, using Word2Vec yielded significant results, while ConceptNet Numberbatch did not, suggesting that how we computationally represent "meaning" can change the outcome of the study.

Future Work could involve:

  • Testing these metrics on even more complex, multi-object scenes.
  • Applying these insights to improve Human-Computer Interaction (HCI) or Augmented Reality (AR) systems that need to "know" what a user is likely to look at next.
  • Exploring how these metrics change across different languages or cultural contexts.

Summary updated

Abstract

Semantic relevance metrics can capture both the inherent semantics of individual objects and their relationships to other elements within a visual scene. Numerous previous research has demonstrated that these metrics can influence human visual processing. However, these studies often did not fully account for contextual information or employ the recent deep learning models for more accurate computation. This study investigates human visual perception and processing by introducing the metrics of contextual semantic relevance. We evaluate semantic relationships between target objects and their surroundings from both vision-based and language-based perspectives. Testing a large eye-movement dataset from visual comprehension, we employ state-of-the-art deep learning techniques to compute these metrics and analyze their impacts on fixation measures on human visual processing through advanced statistical models. These metrics could also simulate top-down and bottom-up processing in visual perception. This study further integrates vision-based and language-based metrics into a novel combined metric, addressing a critical gap in previous research that often treated visual and semantic similarities separately. Results indicate that all metrics could precisely predict fixation measures in visual perception and processing, but with distinct roles in prediction. The combined metric outperforms other metrics, supporting theories that emphasize the interaction between semantic and visual information in shaping visual perception/processing. This finding aligns with growing recognition of the importance of multi-modal information processing in human cognition. These insights enhance our understanding of cognitive mechanisms underlying visual processing and have implications for developing more accurate computational models in fields such as cognitive science and human-computer interaction.
BibTeX
@misc{sun2024roles,
  author = {Kun Sun and Wang Rong},
  title = {{The Roles of Contextual Semantic Relevance Metrics in Human Visual Processing}},
  year = {2024},
  eprint = {2410.09921},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.09921},
}
Figure