Exploration of complex visual feature spaces for object perception

Daniel D. Leeds, John A. Pyles, Michael J. TarrFrontiers Comput. Neurosci 2014 · OpenReview

TL;DR

This paper introduces a novel real-time fMRI approach to explore how the human brain represents complex objects. By dynamically selecting visual stimuli based on brain activity, they discovered that specific brain regions are selective for both holistic object shapes and their component parts, as well as surface textures. This method offers a more efficient way to map out the intricate feature spaces used in object perception.

What Problem Does This Paper Address?

Understanding how our brains recognize objects, especially at the "mid- and high-level" stages in the ventral visual pathway, remains a significant challenge. While we know that information is processed hierarchically, increasing in complexity, the exact nature of the intermediate features that bridge simple visual elements (like edges) to complete object recognition (like "car" or "face") is still elusive. Traditional methods often rely on pre-defined feature models and stimuli, which can be inefficient when exploring the vast and complex feature spaces relevant to real-world objects. This paper aims to overcome this limitation by developing a more adaptive and data-driven approach to probe these feature spaces directly within the human brain.

Methods

The core innovation here is a real-time fMRI stimulus selection method. Instead of presenting a fixed set of images, the system dynamically chooses the next image to display based on the brain's immediate response (measured by fMRI's Blood-Oxygen-Level Dependent, or BOLD, signal) to previously shown stimuli. This "search" process is guided by computational models of visual features:

  • SIFT (Scale-Invariant Feature Transform): A well-established computer vision technique used to represent real-world objects. The researchers created a multi-dimensional feature space based on SIFT descriptors.
  • Fribble Objects: Synthetic, parametrically controlled 3D objects composed of interchangeable parts. These were used to create feature spaces where specific dimensions corresponded to variations in object components (e.g., head shape, leg type).

The real-time search algorithm, inspired by simulated annealing, aims to find stimulus locations within these feature spaces that elicit maximal BOLD responses in pre-selected regions of interest (ROIs) within the ventral visual cortex. This allows for a more efficient exploration of the neural selectivity landscape.

Datasets

The study utilized two main types of visual stimuli:

  • Real-world objects: A dataset of 400 distinct color object photos from the Hemera Photo Objects dataset, covering categories like mammals, human-forms, cars, and containers. These were organized into a SIFT-based feature space.
  • Synthetic "Fribble" objects: Images generated from a library of synthetic Fribbles, which are composed of colored, textured geometric shapes. These were organized into feature spaces specific to four different Fribble classes.

No new benchmark datasets were introduced; rather, existing image sets were leveraged and embedded into novel feature spaces for exploration.

Key Insights and Ideas

  • Dynamic Exploration of Feature Space: The real-time fMRI approach is a powerful tool for efficiently exploring complex, high-dimensional visual feature spaces that are difficult to cover with static stimulus sets.
  • Holistic and Component Selectivity: Brain regions are not just selective for entire objects but also for specific combinations of component features and even surface properties.
  • High-Level "Local Inhibition": Similar to findings in earlier visual processing stages (like edge detection), specific brain regions showed suppressed responses to stimuli that were only slightly different from those that elicited strong responses. This suggests a mechanism for sharpening neural representations.
  • SIFT vs. Fribble Space Effectiveness: The Fribble objects, with their more controlled and structured feature spaces, generally led to more consistent and interpretable search results compared to the SIFT-based space for real-world objects. This highlights the importance of how feature spaces are constructed for effective neural exploration.

Key Results

  • The real-time search method successfully identified brain regions selective for a variety of visual properties, including holistic object shape, component parts, and surface texture.
  • In several instances, stimuli that were close to each other in the defined feature space elicited responses at opposite extremes (high vs. low activation) in the targeted brain regions, providing evidence for high-level "local inhibition."
  • The Fribble object searches, due to their more constrained and interpretable feature spaces, showed better convergence and consistency compared to the SIFT-based searches for real-world objects.
  • While the SIFT space for real-world objects proved challenging to interpret with only four dimensions, it did reveal some general patterns related to coarse shape and surface properties.

Limitations and Future Work

  • SIFT Space Limitations: The use of only four dimensions derived from Multi-Dimensional Scaling (MDS) of the SIFT space for real-world objects was insufficient to capture the full complexity of visual properties, leading to less interpretable results and lower search convergence. The authors suggest that more dimensions or a better-fitted feature set would be needed.
  • Search Convergence: Convergence (where the search consistently identifies the same preferred stimuli) was not always achieved, particularly for the SIFT-based searches. This is attributed to the novelty of the methods, limitations in the number of stimulus display trials, and potentially suboptimal search parameters. Further methodological development is anticipated to improve convergence rates.
  • Potential for Adaptation/Learning: While the Fribble object results suggest adaptation effects might not have been a major confound, the authors acknowledge that repeated exposures across sessions could have influenced interpretations.
  • Future Directions: The authors propose exploring alternative feature spaces, potentially derived from unsupervised statistical learning or combining SIFT with other methods. They also suggest that as these real-time fMRI techniques mature, they will become increasingly valuable for mapping fine-grained neural representations of visual stimuli.

Summary updated

Abstract

The mid- and high-level visual properties supporting object perception in the ventral visual pathway are poorly understood. In the absence of well-specified theory, many groups have adopted a data-driven approach in which they progressively interrogate neural units to establish each unit’s selectivity. Such methods are challenging in that they require search through a wide space of feature models and stimuli using a limited number of samples. To more rapidly identify higher-level features underlying human cortical object perception, we implemented a novel method in which visual stimuli are selected in real-time based on BOLD responses to recently shown stimuli. This work was inspired by earlier primate physiology work, in which neural selectivity for mid-level features in IT was characterized using a simple parametric approach (Hung 2012). To extend such work to human neuroimaging, we used natural and synthetic object stimuli embedded in feature spaces constructed on the basis of the complex visual properties of the objects themselves. During fMRI scanning, we employed a real-time search method to control continuous stimulus selection within each image space. This search was designed to maximize neural responses across a pre-determined 1 cm3 brain region within ventral cortex. To assess the value of this method for understanding object encoding, we examined both the behavior of the method itself and the complex visual properties the method identified as reliably activating selected brain regions. We observed: 1) Regions selective for both holistic and component object features and for a variety of surface properties; 2) Object stimulus pairs near one another in feature space that produce responses at the opposite extremes of the measured activity range. Together, these results suggest that real-time fMRI methods may yield more widely informative measures of selectivity within the broad classes of visual features associated with cortical object representation.
BibTeX
@inproceedings{leeds2014exploration,
  author = {Daniel D. Leeds and John A. Pyles and Michael J. Tarr},
  title = {{Exploration of complex visual feature spaces for object perception}},
  booktitle = {Frontiers Comput. Neurosci},
  year = {2014},
  url = {https://openreview.net/forum?id=5t1Hs78Qh3},
}

Loading related content...

Figure