Active sensing in the categorization of visual patterns

Scott Cheng‐Hsin Yang, Máté Lengyel, Daniel M. WolperteLife 2016 · DOI

TL;DR

Humans actively scan visual scenes to gather information, not just randomly. This study shows that our eye movements are surprisingly efficient, aiming to maximize information gain for categorization tasks. While not perfectly optimal, our "active sensing" strategy is about 70% efficient, with inefficiencies stemming from perceptual noise and motor inaccuracies rather than poor planning.

What Problem Does This Paper Address?

When we look at a scene, our eyes don't just wander; they actively seek out specific information. But how "smart" is this process? Do we pick the best spots to look next to figure out what we're seeing, or is it more of a hit-or-miss affair? Previous studies often focused on simple tasks like visual search, where the target is usually in one place. This paper tackles a more complex problem: how do we actively scan a scene to categorize abstract patterns, where information is spread out and requires integrating clues from multiple locations? The authors want to understand if our eye movements are optimized for this kind of task and where any inefficiencies might lie.

Methods

The researchers developed a novel "gaze-contingent paradigm." Participants looked at images that were initially masked, and only a small aperture of the image was revealed at their fixation point. This allowed them to control exactly what information the participant received at each "look." They then developed a sophisticated Bayesian active sensor (BAS) algorithm. This algorithm acts as an "ideal observer," calculating the optimal next fixation location that would maximize information gain for categorizing the image. By comparing the BAS algorithm's predictions to actual human eye movements, they could quantify how efficient human "active sensing" is. They also used an "ideal observer model" to estimate the information gained from different revealing strategies.

Datasets

The study used synthesized images of three types: patchy, horizontal stripy, and vertical stripy. These images were generated from a flexible statistical model to ensure that categorization relied on higher-order statistical information rather than simple features. Participants were trained on these image types until they could perfectly categorize full images. The experiment then used these synthesized images in the gaze-contingent task.

Key Insights and Ideas

  • Active Sensing is Key: Our eye movements aren't random; they're a deliberate strategy to gather information efficiently for tasks like categorization.
  • Maximizing Information Gain: Participants' scan paths are guided by a goal to maximize the information gained at each fixation, aiming to reduce uncertainty about the image category.
  • Suboptimality is Not (Just) Bad Planning: The 30% inefficiency in human eye movements isn't primarily due to poor decision-making about where to look. Instead, it's largely attributed to perceptual noise (imperfect vision) and motor inaccuracies (imprecise eye movements).
  • Abstract Categories Matter: This active sensing strategy is crucial for tasks requiring the integration of information from multiple locations to understand abstract categories, going beyond simple visual search.

Key Results

  • Participants' categorization performance improved significantly with more revealed locations, demonstrating successful information integration.
  • Their eye movement patterns were strongly dependent on the image type, indicating an active sensing strategy.
  • When comparing human performance to the BAS algorithm, participants were about 70% efficient in selecting their fixation locations.
  • The BAS algorithm, when simulating human-like perception noise and motor inaccuracies, closely predicted participants' revealing patterns.
  • Participants' active scan paths were significantly more efficient (1.8 to 2.9 times) than heuristic strategies that didn't fully incorporate closed-loop Bayesian updating.

Limitations and Future Work

  • Naturalistic Stimuli: The study used synthesized images with stationary statistics. Future work could explore more complex, naturalistic scenes to see if efficiency changes.
  • Task Complexity: While more complex than visual search, the task still involved abstract categories. Extending this to even higher-level cognitive tasks (e.g., judging intentions) is a future direction.
  • Voluntary Eye Movements: The study focused on voluntary saccades. Future research could investigate the role of involuntary movements like microsaccades and drift in extracting information.
  • Learning: While efficiency remained stable over sessions, suggesting minimal learning was required for this specific task, exploring learning in more complex scenarios is an open question.

Summary updated

Abstract

Interpreting visual scenes typically requires us to accumulate information from multiple locations in a scene. Using a novel gaze-contingent paradigm in a visual categorization task, we show that participants' scan paths follow an active sensing strategy that incorporates information already acquired about the scene and knowledge of the statistical structure of patterns. Intriguingly, categorization performance was markedly improved when locations were revealed to participants by an optimal Bayesian active sensor algorithm. By using a combination of a Bayesian ideal observer and the active sensor algorithm, we estimate that a major portion of this apparent suboptimality of fixation locations arises from prior biases, perceptual noise and inaccuracies in eye movements, and the central process of selecting fixation locations is around 70% efficient in our task. Our results suggest that participants select eye movements with the goal of maximizing information about abstract categories that require the integration of information from multiple locations.
BibTeX
@inproceedings{yang2016active,
  author = {Scott Cheng‐Hsin Yang and Máté Lengyel and Daniel M. Wolpert},
  title = {{Active sensing in the categorization of visual patterns}},
  booktitle = {eLife},
  year = {2016},
  doi = {10.7554/elife.12215},
}

Loading related content...

Figure