The Scope and Limits of Fine-Grained Image and Category Information in the Ventral Visual Pathway

Markus W. Badwal, Johanna Bergmann, Johannes Roth, Christian F. Doeller, Martin N. Hebart

Abstract

Humans can easily abstract incoming visual information into discrete semantic categories. Previous research employing functional MRI (fMRI) in humans has identified cortical organizing principles that allow not only for coarse-scale distinctions such as animate versus inanimate objects but also more fine-grained distinctions at the level of individual objects. This suggests that fMRI carries rather fine-grained information about individual objects. However, most previous work investigating fine-grained category representations either additionally included coarse-scale category comparisons of objects, which confounds fine-grained and coarse-scale distinctions, or only used a single exemplar of each object, which confounds visual and semantic information. To address these challenges, here we used multisession human fMRI (female and male) paired with a broad yet homogenous stimulus class of 48 terrestrial mammals, with 2 exemplars per mammal. Multivariate decoding and representational similarity analysis (RSA) revealed high image-specific reliability in low- and high-level visual regions, indicating stable representational patterns at the image level. In contrast, analyses across exemplars of the same animal yielded only small effects in the lateral occipital complex (LOC), indicating rather subtle category effects in this region. Variance partitioning with a deep neural network and shape model showed that across exemplar effects in EVC were largely explained by low-level visual appearance, while representations in LOC appeared to also contain higher category-specific information. These results suggest that representations typically measured with fMRI are dominated by image-specific visual or coarse-grained category information but indicate that commonly employed fMRI protocols may reveal subtle yet reliable distinctions between individual objects.

Introduction

Neuroscientists have long established that the ventral visual pathway organizes information into broad categories, such as the distinction between animate and inanimate objects. However, the extent to which functional MRI (fMRI) can capture "fine-grained" distinctions—such as differentiating between a zebra and a horse—remains a subject of debate. Previous research often suffered from two primary confounds: coarse-scale category comparisons (e.g., mammals vs. insects) and the use of single exemplars, which conflates semantic category with specific visual features.

This study addresses these challenges by investigating whether the human brain represents the abstract concept of a specific animal species independently of the specific image used to depict it. By focusing on a highly homogenous group of 48 terrestrial mammals, the researchers aimed to isolate fine-grained category information from low-level visual properties.

Methods and Materials

The study utilized a multisession fMRI design with 15 adult participants to maximize the signal-to-noise ratio.

Stimulus Set

The primary dataset consisted of 96 custom-curated images of 48 terrestrial mammal species, with two distinct exemplars per species. All images were controlled for the presence of both faces and bodies to eliminate the "face-body" distinction as a confounding variable.

The stimulus set consisted of 48 terrestrial mammal species, each represented by two distinct exemplars to decouple visual features from semantic categories.

Experimental Design

Participants performed a "color context" task across three scanning sessions. During the memorization phase, images were presented on green or red backgrounds. In the subsequent retrieval phase, participants viewed the same images on a neutral gray background and were tasked with recalling the original color context.

The experimental paradigm included a memorization task and a retrieval task, designed to maintain participant engagement while measuring neural responses to the stimuli.

Analysis Techniques

The researchers employed several advanced analytical techniques:

  • Multivariate Pattern Analysis (MVPA): Linear Support Vector Machines (SVM) were used to decode both image-specific and cross-exemplar (category-specific) information.
  • Representational Similarity Analysis (RSA): Neural activity patterns were compared against computational models, including AlexNet (for visual features), ShapeComp (for shape similarity), and semantic models like THINGS and CSLB.
  • Variance Partitioning: This statistical approach allowed the team to isolate the unique contribution of semantic information by controlling for low-level and mid-level visual features.

Key Results

The Dominance of Image-Specific Information

The results revealed that the strongest neural signals in the ventral stream are image-specific. Decoding accuracy for specific photographs was remarkably high, reaching approximately 89% in the Early Visual Cortex (EVC) and 76% in the Lateral Occipital Complex (LOC). This suggests that even in high-level visual areas, the brain's response is heavily influenced by the specific visual properties of an image.

LOC as a Hub for Fine-Grained Categorization

While image-specific effects dominated, the study found subtle but reliable category-specific information in the LOC. Unlike the EVC, where "category" effects were largely explained by visual similarity between exemplars, the LOC maintained significant decoding accuracy (~56%) even for the most visually dissimilar pairs of exemplars. This indicates that the LOC represents fine-grained categories in a way that transcends simple visual appearance.

Semantic Representations and the "Anterior Silence"

RSA correlations showed that neural patterns in the posterior fusiform (pFus) aligned with human-perceived semantic similarity (THINGS and CSLB models), even after accounting for visual features. Interestingly, more anterior regions like the Hippocampus and Perirhinal Cortex showed very weak signals for these fine-grained distinctions. This "anterior silence" suggests that current fMRI protocols may lack the sensitivity to capture the highly abstract representations typically associated with these areas.

Discussion and Conclusion

This research underscores the capacity of fMRI to uncover fine-grained visual information while highlighting the overwhelming influence of image-specific features. The findings pinpoint the LOC as a critical region for bridging the gap between visual appearance and semantic categorization.

Limitations and Future Directions

The use of only two exemplars per category may limit the degree of abstraction possible in the analysis. Future work could benefit from a more diverse set of poses and contexts for each species. Additionally, the lack of strong effects in the anterior temporal lobe (ATL) might be addressed by using 7T fMRI or Magnetoencephalography (MEG) to improve spatial resolution and capture temporal dynamics, respectively.

Ultimately, this study provides a clearer picture of how the ventral visual pathway balances the processing of specific visual inputs with the need to categorize the world into meaningful semantic units.

Summary updated

Abstract

Humans can easily abstract incoming visual information into discrete semantic categories. Previous research employing functional MRI (fMRI) in humans has identified cortical organizing principles that allow not only for coarse-scale distinctions such as animate versus inanimate objects but also more fine-grained distinctions at the level of individual objects. This suggests that fMRI carries rather fine-grained information about individual objects. However, most previous work investigating fine-grained category representations either additionally included coarse-scale category comparisons of objects, which confounds fine-grained and coarse-scale distinctions, or only used a single exemplar of each object, which confounds visual and semantic information. To address these challenges, here we used multisession human fMRI (female and male) paired with a broad yet homogenous stimulus class of 48 terrestrial mammals, with two exemplars per mammal. Multivariate decoding and representational similarity analysis revealed high image-specific reliability in low- and high-level visual regions, indicating stable representational patterns at the image level. In contrast, analyses across exemplars of the same animal yielded only small effects in the lateral occipital complex (LOC), indicating rather subtle category effects in this region. Variance partitioning with a deep neural network and shape model showed that across-exemplar effects in the early visual cortex were largely explained by low-level visual appearance, while representations in LOC appeared to also contain higher category-specific information. These results suggest that representations typically measured with fMRI are dominated by image-specific visual or coarse-grained category information but indicate that commonly employed fMRI protocols may reveal subtle yet reliable distinctions between individual objects.
BibTeX
@article{badwal2024scope,
  author = {Markus W. Badwal and Johanna Bergmann and Johannes Roth and Christian F. Doeller and Martin N. Hebart},
  title = {{The Scope and Limits of Fine-Grained Image and Category Information in the Ventral Visual Pathway}},
  journal = {Journal of Neuroscience},
  year = {2024},
  doi = {10.1523/jneurosci.0936-24.2024},
}
Figure