Optimal visual search based on a model of target detectability in natural images

Shima Rashidi, Krista A. Ehinger, Andrew Turpin, Lars KulikNeurIPS 2020 · OpenReview

Researchers have developed a new model that predicts how easily a human can detect a target in a complex natural image, even when it’s in their peripheral vision. By combining deep learning features with a Bayesian "ideal observer" framework, the model successfully simulates human eye movement patterns during visual search tasks.

When you look for your keys on a cluttered table, your brain doesn't process the whole scene at once. Humans have "foveated" vision: we see high detail at the center of our gaze (the fovea) and much less detail in the periphery. To find something, we move our eyes in a sequence of fixations.

In computer science, most models of this process rely on "saliency"—identifying what parts of an image naturally "pop out." However, saliency doesn't account for detectability, which is the actual statistical likelihood of seeing a specific target against a specific background. While we can easily calculate detectability for simple shapes on static noise, doing so for complex objects in natural, textured scenes is incredibly difficult. This paper addresses the challenge of automatically estimating target detectability in natural images to predict optimal, human-like search paths.

A Multi-Stage Detectability Pipeline

To bridge this gap, the authors propose a multi-stage pipeline to compute a "detectability map" that guides a simulated searcher.

The proposed multi-stage pipeline for computing a detectability map using a pretrained CNN and eccentricity-specific classifiers.
  1. Feature Extraction: The researchers use a Convolutional Neural Network (CNN)—specifically AlexNet pre-trained on ImageNet—to extract high-level visual features from image patches. They argue that CNN hierarchies effectively mimic the feed-forward processing of the human visual system.
  2. Classification: These extracted features are fed into a logistic regression classifier trained to distinguish between "target-present" and "target-absent" patches.
  3. Modeling the Periphery (Eccentricity): To simulate the loss of detail in peripheral vision, the authors use a "feature pooling" approach. They increase the size of the image patches processed by the CNN as the distance from the center of the gaze increases. This mimics how human neurons have larger receptive fields in the periphery.
  4. The Bayesian Ideal Observer: The resulting detectability estimates are plugged into a Bayesian framework. This "ideal observer" updates its internal map of where the target might be after every "look" and chooses the next fixation point that maximizes the information gained, aiming to find the target in the fewest moves possible.

Key Insights: Foveation and Biological Proxies

The study confirms that complex features from deep networks are far better at predicting human detectability than hand-crafted features like luminance histograms or "Textons."

Crucially, the authors demonstrate that foveation is more than just blur. Instead of simply blurring the periphery (a common approach in older models), modeling the periphery as a change in "patch size" (feature pooling) more accurately reflects the "crowding" effect in human vision. Objects become harder to distinguish not just because they are blurry, but because their features blend together with surrounding textures.

Illustration of how target discriminability decreases as eccentricity increases, modeling the drop in peripheral vision performance.

Furthermore, the model highlights that detectability is task-specific. It isn't a fixed number for a target; it's a dynamic relationship between the target's features, the background's complexity, and the observer's current point of fixation.

Datasets and Results

The model was trained and evaluated using several datasets, including the ETH and Cimpoi Texture Datasets for backgrounds, ImageNet for CNN pre-training, and custom human eye-tracking data collected from participants performing detection tasks.

  • Model Accuracy: The combination of AlexNet and Logistic Regression achieved the lowest Mean Squared Error (MSE = 0.0978) when compared to human detectability data, significantly outperforming deeper networks like VGG19 or simpler hand-crafted models.
  • Human-Like Search: When the detectability model was used to drive the Bayesian searcher, the number of fixations required to find a target closely matched the performance of human observers across 18 different backgrounds.
  • The "Oracle" Comparison: The model performed nearly as well as an "oracle" model that was given actual human detection data, suggesting the automated pipeline is a highly effective substitute for expensive psychophysical testing.

Limitations and Future Work

While highly effective, the study focused primarily on "textured" backgrounds that are relatively homogeneous. Real-world search often involves heterogeneous scenes where different regions have wildly different properties, introducing "contextual priors" (like knowing a toaster is usually on a counter) that this model does not yet capture. Additionally, humans often perform slightly worse than the "ideal" model, likely due to internal noise or distractions.

Looking forward, the authors suggest this work could be applied to augmented reality (AR) systems, potentially directing a driver's attention to hazards they might have missed in their peripheral vision by understanding exactly what is and isn't detectable in their current gaze state.

Summary updated

Abstract

To analyse visual systems, the concept of an ideal observer promises an optimal response for a given task. Bayesian ideal observers can provide optimal responses under uncertainty, if they are given the true distributions as input. In visual search tasks, prior studies have used signal to noise ratio (SNR) or psychophysics experiments to set the distributional parameters for simple targets on backgrounds with known patterns, however these methods do not easily translate to complex targets on natural scenes. Here, we develop a model of target detectability in natural images to estimate the parameters of target-present and target-absent distributions for a visual search task. We present a novel approach for approximating the foveated detectability of a known target in natural backgrounds based on biological aspects of human visual system. Our model considers both the uncertainty about target position and the visual system's variability due to its reduced performance in the periphery compared to the fovea. Our automated prediction algorithm uses trained logistic regression as a post processing phase of a pre-trained deep neural network. Eye tracking data from 12 observers detecting targets on natural image backgrounds are used as ground truth to tune foveation parameters and evaluate the model, using cross-validation. Finally, the model of target detectability is used in a Bayesian ideal observer model of visual search, and compared to human search performance.
BibTeX
@inproceedings{rashidi2020optimal,
  author = {Shima Rashidi and Krista A. Ehinger and Andrew Turpin and Lars Kulik},
  title = {{Optimal visual search based on a model of target detectability in natural images}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=vFiW05xXfRv},
}
Figure