TL;DR
This paper introduces a novel way to quantify "object recognition uncertainty" in images and shows it's a surprisingly strong predictor of where people look, especially during free viewing. While target features dominate visual search, this uncertainty metric outperforms traditional saliency models for free viewing, suggesting a deeper cognitive process at play.
What Problem Does This Paper Address?
Understanding how our attention is guided is a fundamental question in vision science. For decades, researchers have identified several key factors influencing where we look: bottom-up saliency (visually striking features), center bias (a tendency to look at the image center), and target features (when searching for a specific item). However, most studies focus on one or two of these factors, making it difficult to understand their relative importance and how they interact across different tasks like free viewing (just looking around) and visual search (looking for something specific). This paper aims to address this by simultaneously modeling and comparing the contributions of these factors, with a particular focus on a less-explored factor: object recognition uncertainty.
Methods
The core innovation of this paper is a new method for quantifying object recognition uncertainty. They leverage Mask R-CNN, a popular instance segmentation model, to generate "object proposals" (potential object bounding boxes). For each proposal, they count the number of object categories that the model considers plausible. The more categories that compete for a given proposal, the higher the "object recognition uncertainty." This uncertainty is then used to create an "uncertainty map."
They also implemented models for:
- Bottom-up saliency: Using the Graph-Based Visual Saliency (GBVS) model, which relies solely on low-level image features like contrast.
- Center bias: A simple 2D Gaussian centered on the image.
- Target features: For visual search tasks, they used Mask R-CNN again, but this time with a high confidence threshold to identify the specific target object.
To quantify the predictive power of these factors, they employed Generalized Linear Mixed Models (GLMMs). This statistical approach allows them to assess the unique contribution of each factor (priority map) to predicting human fixations, while accounting for variability across subjects and images.
Datasets
The study utilized two benchmarked datasets:
- OSIE dataset: Used for free-viewing tasks, consisting of 700 images with annotated fixations from 15 participants. Images containing humans or animals were excluded to avoid known biases.
- COCO-Search18 dataset: Used for visual search tasks, containing approximately 300,000 fixations from 10 people searching for 18 different target-object categories across 6202 images. This dataset includes both target-present (TP) and target-absent (TA) trials.
Key Insights and Ideas
- Object recognition uncertainty is a powerful driver of attention, especially in free viewing: This is the most striking finding. The paper proposes that when our visual system is unsure about what an object is (i.e., multiple categories could fit), it directs attention to resolve this ambiguity.
- Target features dominate in target-present search, but are weak in target-absent search: As expected, when a specific target is present, features of that target are the primary driver of attention. However, surprisingly, these target features play a minimal role when the target is absent, suggesting that attention in TA search is more exploratory and driven by other factors.
- Saliency is more predictive of search than free viewing, but uncertainty is king for free viewing: While bottom-up saliency models are generally good at predicting fixations in visual search, the object recognition uncertainty model surprisingly outperforms saliency for free viewing.
- The "need to recognize objects" is a fundamental driver of attention: The authors hypothesize that the core function of attention is to facilitate object recognition, and uncertainty in this process naturally draws our gaze.
Key Results
- In free viewing, object recognition uncertainty was consistently a better predictor of fixations than saliency or center bias, except for the very first fixation where center bias was strongest.
- In target-present (TP) search, target features were overwhelmingly the most predictive factor, dominating all others.
- In target-absent (TA) search, target features were among the least predictive factors. Instead, object recognition uncertainty and saliency showed a stronger influence, with saliency being slightly more predictive.
- The paper also compared their proposed object recognition uncertainty model to two alternative definitions (label-entropy and pixel-wise uncertainty) using the Normalized Scanpath Saliency (NSS) metric. Their original model achieved the highest NSS scores (0.91 for pixel-wise, 1.35 for their own formulation) compared to label-entropy (0.74), suggesting their method is more psychologically meaningful for predicting free-viewing fixations.
Limitations and Future Work
- Interpretability of "meaning maps": The authors note that their formulation of object recognition uncertainty is computer-vision-based and applicable to many images. However, they acknowledge that future work could explore how this relates to "meaning maps" (which are often hand-labeled) and whether meaning itself plays a role beyond object recognition.
- TA search complexity: The unexpected weakness of target features in TA search might be due to the specific experimental setup. The authors suggest that future research could investigate whether this holds true in more constrained search contexts.
- Deep learning models: The paper intentionally avoids comparing their model directly to complex deep learning models trained on fixation data, as these models often combine multiple factors in an uninterpretable way. However, they acknowledge that these models exist and could be a point of comparison in future work, provided their "black box" nature can be overcome.
Summary updated