A mid-level organization of the ventral stream

Bria Long, Chen-Ping Yu, Talia KonklebioRxiv 2017 · DOI

TL;DR

This paper introduces "texforms," abstract visual stimuli that retain mid-level features but are unrecognizable. They demonstrate that these texforms are sufficient to drive large-scale animacy and object-size organizations in the ventral visual stream, suggesting that these high-level categories are organized by relatively primitive mid-level features, not just by recognizing complete objects.

What Problem Does This Paper Address?

For decades, researchers have debated what kind of information drives the organization of the ventral visual stream, the part of the brain responsible for object recognition. Specifically, it's unclear whether this organization is based on recognizing complete objects (e.g., "this is a dog") or on more fundamental, mid-level visual features (e.g., "this has fur and four legs"). Previous studies often used recognizable objects, making it difficult to disentangle the effects of object recognition from the underlying visual features. This paper aims to address this by using stimuli that preserve mid-level features but prevent object recognition, allowing researchers to isolate the role of these features.

Methods

The core of this study is the creation and use of "texforms." These are abstract stimuli generated using a texture synthesis algorithm that preserves mid-level texture and form information from original images but removes recognizable contours and parts, rendering them unrecognizable. The researchers then used fMRI to measure brain activity in participants viewing these texforms and their original counterparts. They analyzed these responses using several methods:

  • Topographical Analysis: Mapping where in the brain different categories (animacy, object size) were represented.
  • Preference Map Correlations: Quantifying the similarity between brain activity patterns evoked by texforms and original images.
  • Posterior-to-Anterior Gradient Analysis: Examining how representations change from the back to the front of the ventral stream.
  • Conjunction Analysis: Identifying brain regions that show consistent category preferences regardless of the stimulus's position in the visual field.
  • Predictive Modeling: Using various computational models (image statistics, deep convolutional neural networks, behavioral ratings of curvature) to predict the observed neural patterns.

Datasets

The study utilized a novel dataset of 240 images: 120 original images of animals and objects of different sizes, and 120 corresponding "texforms" generated from these originals. The texforms were specifically selected to be unrecognizable at a basic level. Additionally, behavioral data from norming studies (animacy/size classification, curvature ratings) were used to characterize the texforms and originals.

Key Insights and Ideas

  • Unrecognizable stimuli can drive high-level organization: The most striking finding is that abstract texforms, which participants couldn't identify, were still sufficient to elicit large-scale animacy and object-size organizations in the ventral stream. This strongly suggests that these organizations are built upon mid-level features, not just object recognition.
  • Mid-level features are key, not just low-level statistics: Models based on simple image statistics (like Gabor filters) failed to predict neural responses, while models based on intermediate layers of deep convolutional neural networks (CNNs) and perceived curvature were much more successful. This highlights the importance of intermediate-complexity features.
  • Curvature as a unifying principle: Perceived curvature (or the "boxy-curvy" axis) emerged as a significant predictor of neural responses, aligning with broad category distinctions like animacy and object size. This suggests curvature might be a fundamental organizational principle.
  • Ventral stream is not strictly hierarchical: The study found that texform-based topographies extended throughout the ventral stream, including areas typically considered "high-level," challenging a simple hierarchical model where early areas process simple features and later areas process complex objects.

Key Results

  • Texforms elicited animacy and object-size topographies across the entire occipito-temporal cortex, showing significant correlations (r = .76 for animacy, r = .43 for object size) with the topographies evoked by original, recognizable images.
  • These texform-based organizations were tolerant to changes in visual field position, suggesting they are not simply retinotopic biases.
  • Intermediate layers of deep CNNs and perceived curvature ratings best predicted the neural similarity structure of texforms, explaining a substantial portion of the variance (e.g., CNN Layer 4/5: TA ≈ .32-.34; Curvature: TA = .28).
  • Original images elicited stronger categorical responses than texforms, but this difference was consistent across the posterior-to-anterior gradient, not confined to anterior regions.

Limitations and Future Work

  • Defining Curvature: The authors acknowledge that "curvature" is difficult to define computationally and has been operationalized in different ways across studies, making it unclear if findings derive from a single underlying construct.
  • Direction of Causality: The study establishes a link between mid-level features and high-level categories but cannot determine the direction of causality (i.e., whether category biases influence curvature perception or vice versa).
  • Texform vs. Original Differences: While texforms drove similar organizations, original images elicited stronger responses. The authors speculate this might be due to additional category-specific or generic visual features (like 3D structure and contours) present in original images.
  • Future Work: The authors suggest further research into the minimal amount of categorization training needed for CNN units to predict categorical neural responses and exploring the directionality of the relationship between mid-level features and high-level categories.

Summary updated

Abstract

ABSTRACT Human object-selective cortex shows a large-scale organization characterized by the high-level properties of both animacy and object-size. To what extent are these neural responses explained by primitive perceptual features that distinguish animals from objects and big objects from small objects? To address this question, we used a texture synthesis algorithm to create a novel class of stimuli— texforms —which preserve some mid-level texture and form information from objects while rendering them unrecognizable. We found that unrecognizable texforms were sufficient to elicit the large-scale organizations of object-selective cortex along the entire ventral pathway. Further, the structure in the neural patterns elicited by texforms was well predicted by curvature features and by intermediate layers of a deep convolutional neural network, supporting the mid-level nature of the representations. These results provide clear evidence that a substantial portion of ventral stream organization can be accounted for by coarse texture and form information, without requiring explicit recognition of intact objects. SIGNIFICANCE STATEMENT While neural responses to object categories are remarkably systematic across human visual cortex, the nature of these responses been hotly debated for the past 20 years. In this paper, a new class of stimuli (“texforms”) is used to examine how mid-level features contribute to the large-scale organization of the ventral visual stream. Despite their relatively primitive visual appearance, these unrecognizable texforms elicited the entire large-scale organizations of the ventral stream by animacy and object size. This work demonstrates that much of ventral stream organization can be explained by relatively primitive mid-level features, without requiring explicit recognition of the objects themselves.
BibTeX
@misc{long2017mid,
  author = {Bria Long and Chen-Ping Yu and Talia Konkle},
  title = {{A mid-level organization of the ventral stream}},
  year = {2017},
  doi = {10.1101/213934},
  howpublished = {bioRxiv},
}
Figure