ViSioNS: Visual Search in Natural Scenes Benchmark

Fermín Travi, Gonzalo Ruarte, Gastón Bujia, Juan E. KamienkowskiNeurIPS 2022

Searching for your keys on a cluttered desk or spotting a specific car in a busy parking lot feels intuitive to humans, but for artificial intelligence, "visual search" in natural scenes remains a complex puzzle. While the computer vision community has established excellent benchmarks for saliency (predicting where people look when freely viewing an image), the field of goal-directed search has long been a fragmented "Wild West."

A new paper from researchers at the University of Buenos Aires, titled ViSioNS, aims to change this landscape by providing the first unified framework to compare how well AI models actually mimic human searching behavior.

In the world of computer vision, there are many architectural paradigms for building a "searcher." Some utilize deep convolutional neural networks (CNNs), others employ Reinforcement Learning (RL), and some rely on Bayesian probability.

Historically, research groups tested their models on disparate datasets using custom metrics, making objective comparisons nearly impossible. Furthermore, evaluations often focused solely on search efficiency (how many steps it takes to find a target) rather than similarity (whether the AI's gaze mimics human scanpaths). ViSioNS addresses this evaluation gap by creating a standardized "decathlon" for visual search.

Methods: A Unified Evaluation Pipeline

Rather than simply aggregating data, the authors built a common computational pipeline to ensure fair, apples-to-apples comparisons across different model architectures.

The ViSioNS benchmark pipeline comparing visual search models against human participant data.

Key methodological contributions include:

  • Model Porting: The team ported three major architectural paradigms—IVSN (Zero-shot CNN), IBS (Bayesian), and IRL (Inverse Reinforcement Learning)—into a single Python framework.
  • The nnIBS Hybrid: The authors introduced a novel model, nnIBS, which combines the mathematical rigor of the Ideal Bayesian Searcher with the powerful feature extraction capabilities of modern neural networks.
  • Unified Metrics: The benchmark moves beyond simple "success rates" to include comprehensive similarity metrics:
  • Multi-Match (MM): Measures the spatial and temporal similarity of the entire scanpath.
  • Human Scanpath Prediction (HSP): A fixation-by-fixation test that evaluates whether an AI can predict a human's next gaze location based on their current one.
  • Gold Standard Baseline: They calculated a "human-to-human" similarity score to provide an upper bound of what a perfect, biologically plausible model should achieve.

Standardized Datasets

The benchmark standardizes four diverse, publicly available datasets: Interiors (crowded indoor scenes), Unrestricted (unconstrained images including human figures), MCS (focused on microwaves and clocks), and COCOSearch18 (a large-scale dataset of 18 object categories). Significant preprocessing was performed to standardize fovea sizes and "target found" criteria across all four.

Key Insights: Efficiency vs. Human-Likeness

The ViSioNS benchmark reveals a critical distinction between raw search performance and biological plausibility.

Comparison of scanpaths generated by IVSN, nnIBS, IRL, and human participants on a cluttered desk scene.
  • Efficiency $\neq$ Human-likeness: The IVSN model proved incredibly efficient, finding targets in very few fixations (AUCperf of 0.67 vs. human 0.56). However, its scanpaths did not resemble human behavior. It acts more like a "greedy" optimizer, ignoring the natural constraints and contextual clues of human vision. It failed significantly on information-theoretic metrics like Information Gain (IG).
  • The Bayesian Edge: Models from the IBS family—especially the new nnIBS—were the most consistent with human behavior. Rather than just looking for the target, they maintain an evolving "belief map" of the scene that mirrors human internal logic. nnIBS emerged as the top performer with a relative score of -0.17 (where 0.00 is the Gold Standard), achieving the best balance of efficiency and human-like scanpath prediction.
  • The Context Gap: Humans are highly "context-aware" (e.g., we look for cars on roads, not in the sky). Current AI models still struggle to incorporate this semantic "common sense," often getting distracted by visually similar but contextually impossible objects.

Limitations and Future Work

The authors note that current models are still largely "blind" to the deeper semantic meaning of a scene. While a human immediately understands the "gist" of a kitchen, an AI primarily sees a collection of pixels and local features.

Future advancements will likely require relational reasoning—teaching models that a "fork" is usually near a "plate." By releasing all their code and preprocessed data on GitHub, the ViSioNS authors invite the community to add new models to the leaderboard, paving the way to finally close the gap between machine and human vision.

Summary updated

BibTeX
@inproceedings{travi2022visions,
  author = {Fermín Travi and Gonzalo Ruarte and Gastón Bujia and Juan E. Kamienkowski},
  title = {{ViSioNS: Visual Search in Natural Scenes Benchmark}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
}
Figure