Weighting the factors affecting attention guidance during free viewing and visual search: The unexpected role of object recognition uncertainty

Souradeep Chakraborty, Dimitris Samaras, Gregory J. ZelinskyJournal of Vision 2022 · DOI

TL;DR

This paper introduces a novel way to quantify "object recognition uncertainty" in images and shows it's a surprisingly strong predictor of where people look, especially during free viewing. While target features dominate visual search, this uncertainty metric outperforms traditional saliency models for free viewing, suggesting a deeper cognitive process at play.

What Problem Does This Paper Address?

Understanding how our attention is guided is a fundamental question in vision science. For decades, researchers have identified several key factors influencing where we look: bottom-up saliency (visually striking features), center bias (a tendency to look at the image center), and target features (when searching for a specific item). However, most studies focus on one or two of these factors, making it difficult to understand their relative importance and how they interact across different tasks like free viewing (just looking around) and visual search (looking for something specific). This paper aims to address this by simultaneously modeling and comparing the contributions of these factors, with a particular focus on a less-explored factor: object recognition uncertainty.

Methods

The core innovation of this paper is a new method for quantifying object recognition uncertainty. They leverage Mask R-CNN, a popular instance segmentation model, to generate "object proposals" (potential object bounding boxes). For each proposal, they count the number of object categories that the model considers plausible. The more categories that compete for a given proposal, the higher the "object recognition uncertainty." This uncertainty is then used to create an "uncertainty map."

They also implemented models for:

  • Bottom-up saliency: Using the Graph-Based Visual Saliency (GBVS) model, which relies solely on low-level image features like contrast.
  • Center bias: A simple 2D Gaussian centered on the image.
  • Target features: For visual search tasks, they used Mask R-CNN again, but this time with a high confidence threshold to identify the specific target object.

To quantify the predictive power of these factors, they employed Generalized Linear Mixed Models (GLMMs). This statistical approach allows them to assess the unique contribution of each factor (priority map) to predicting human fixations, while accounting for variability across subjects and images.

Datasets

The study utilized two benchmarked datasets:

  • OSIE dataset: Used for free-viewing tasks, consisting of 700 images with annotated fixations from 15 participants. Images containing humans or animals were excluded to avoid known biases.
  • COCO-Search18 dataset: Used for visual search tasks, containing approximately 300,000 fixations from 10 people searching for 18 different target-object categories across 6202 images. This dataset includes both target-present (TP) and target-absent (TA) trials.

Key Insights and Ideas

  • Object recognition uncertainty is a powerful driver of attention, especially in free viewing: This is the most striking finding. The paper proposes that when our visual system is unsure about what an object is (i.e., multiple categories could fit), it directs attention to resolve this ambiguity.
  • Target features dominate in target-present search, but are weak in target-absent search: As expected, when a specific target is present, features of that target are the primary driver of attention. However, surprisingly, these target features play a minimal role when the target is absent, suggesting that attention in TA search is more exploratory and driven by other factors.
  • Saliency is more predictive of search than free viewing, but uncertainty is king for free viewing: While bottom-up saliency models are generally good at predicting fixations in visual search, the object recognition uncertainty model surprisingly outperforms saliency for free viewing.
  • The "need to recognize objects" is a fundamental driver of attention: The authors hypothesize that the core function of attention is to facilitate object recognition, and uncertainty in this process naturally draws our gaze.

Key Results

  • In free viewing, object recognition uncertainty was consistently a better predictor of fixations than saliency or center bias, except for the very first fixation where center bias was strongest.
  • In target-present (TP) search, target features were overwhelmingly the most predictive factor, dominating all others.
  • In target-absent (TA) search, target features were among the least predictive factors. Instead, object recognition uncertainty and saliency showed a stronger influence, with saliency being slightly more predictive.
  • The paper also compared their proposed object recognition uncertainty model to two alternative definitions (label-entropy and pixel-wise uncertainty) using the Normalized Scanpath Saliency (NSS) metric. Their original model achieved the highest NSS scores (0.91 for pixel-wise, 1.35 for their own formulation) compared to label-entropy (0.74), suggesting their method is more psychologically meaningful for predicting free-viewing fixations.

Limitations and Future Work

  • Interpretability of "meaning maps": The authors note that their formulation of object recognition uncertainty is computer-vision-based and applicable to many images. However, they acknowledge that future work could explore how this relates to "meaning maps" (which are often hand-labeled) and whether meaning itself plays a role beyond object recognition.
  • TA search complexity: The unexpected weakness of target features in TA search might be due to the specific experimental setup. The authors suggest that future research could investigate whether this holds true in more constrained search contexts.
  • Deep learning models: The paper intentionally avoids comparing their model directly to complex deep learning models trained on fixation data, as these models often combine multiple factors in an uninterpretable way. However, they acknowledge that these models exist and could be a point of comparison in future work, provided their "black box" nature can be overcome.

Summary updated

Abstract

The factors determining how attention is allocated during visual tasks have been studied for decades, but few studies have attempted to model the weighting of several of these factors within and across tasks to better understand their relative contributions. Here we consider the roles of saliency, center bias, target features, and object recognition uncertainty in predicting the first nine changes in fixation made during free viewing and visual search tasks in the OSIE and COCO-Search18 datasets, respectively. We focus on the latter-most and least familiar of these factors by proposing a new method of quantifying uncertainty in an image, one based on object recognition. We hypothesize that the greater the number of object categories competing for an object proposal, the greater the uncertainty of how that object should be recognized and, hence, the greater the need for attention to resolve this uncertainty. As expected, we found that target features best predicted target-present search, with their dominance obscuring the use of other features. Unexpectedly, we found that target features were only weakly used during target-absent search. We also found that object recognition uncertainty outperformed an unsupervised saliency model in predicting free-viewing fixations, although saliency was slightly more predictive of search. We conclude that uncertainty in object recognition, a measure that is image computable and highly interpretable, is better than bottom-up saliency in predicting attention during free viewing.
BibTeX
@article{chakraborty2022weighting,
  author = {Souradeep Chakraborty and Dimitris Samaras and Gregory J. Zelinsky},
  title = {{Weighting the factors affecting attention guidance during free viewing and visual search: The unexpected role of object recognition uncertainty}},
  journal = {Journal of Vision},
  year = {2022},
  doi = {10.1167/jov.22.4.13},
}
Figure