Contributions of early and mid-level visual cortex to high-level object categorization

Lily E. Kramer, Yi-Chia Chen, Bria Long, Talia Konkle, Marlene R. CohenbioRxiv 2023 · DOI

TL;DR

This study challenges the prevailing view that high-level object categorization relies solely on later visual processing stages. By recording neural activity in early visual areas (V1 and V4) of monkeys viewing simplified "texform" images, the researchers found that even these early areas contain information about object animacy and real-world size. Crucially, this neural information correlated with human ability to categorize these simplified images, suggesting that early visual processing already begins to disentangle complex object properties.

What Problem Does This Paper Address?

The traditional view in neuroscience posits a hierarchical processing of visual information, where early visual areas (like V1) detect simple features (edges, orientations), and progressively higher areas (like IT cortex) integrate these features into more complex representations necessary for high-level tasks like object recognition and categorization. This implies that abstract categories like "animal" or "object," and properties like "big" or "small," are "entangled" in early visual cortex and only become linearly separable in later stages. However, recent behavioral studies have shown that humans can categorize objects even when only low- and mid-level features are preserved (using "texform" stimuli), raising the question of whether early visual areas already contain such disentangled information. This paper investigates this by directly measuring neural responses in early visual areas to these simplified stimuli.

Methods

The researchers used electrophysiological recordings from populations of neurons in primary visual cortex (V1) and visual area V4 of three rhesus monkeys. They presented two types of stimuli:

  1. Unaltered Images: Standard, recognizable images of animals and objects.
  2. Texforms: Unrecognizable images generated from unaltered images using a texture synthesis algorithm that preserves low- and mid-level image statistics but scrambles higher-level structure.

They employed cross-validated linear decoding to determine if animacy and real-world size could be predicted from the neural population responses. This involved training a linear classifier on a subset of the data and testing its performance on the remaining data. They also conducted cross-decoding analyses, training a decoder on one stimulus type (e.g., texforms) and testing it on the other (e.g., unaltered images) to assess whether the underlying neural representations were similar. Finally, they correlated monkey neural decoding performance with human classification performance on the same texform stimuli to link neural signals to behavior. Temporal analyses were also performed to examine decoding performance across different time windows after stimulus onset.

Datasets

The study utilized a dataset of 480 unique stimuli, comprising 240 unaltered images and their 240 corresponding texform counterparts. These stimuli were drawn from two sources: a stimulus set used in previous work by Long et al. (2018) and a set generated via an accelerated texform method (Deza et al., 2019). The stimuli were categorized into big animals, small animals, big objects, and small objects. Human psychophysical data for texform classification was also used, collected from previous studies.

Key Insights and Ideas

  • Early Visual Cortex Holds High-Level Information: The most striking finding is that even early visual areas (V1 and V4) contain linearly decodable information about animacy and real-world size, challenging the notion that these abstract categories are exclusively processed in later stages.
  • Texforms as a Powerful Tool: The use of texforms proved crucial. By preserving low- and mid-level features while destroying recognizability, they allowed researchers to isolate the contribution of these basic features to higher-level perception.
  • Neural Signals Mirror Human Perception: The strong correlation between the monkeys' neural decoding performance and human observers' ability to classify texforms suggests that the neural representations in V1 and V4 are directly relevant to how we perceive and categorize the world.
  • Untangling Begins Early: The results imply that the process of disentangling complex object properties into distinct neural signals starts much earlier in the visual hierarchy than previously assumed.

Key Results

  • Decodable Animacy and Size: Animacy and real-world size were reliably decoded from V1 and V4 population activity for both unaltered images and texforms.
  • Texform Decoding Success: Critically, animacy was decodable from texforms in V1 and V4, and real-world size was decodable from texforms in V4. While size decoding from V1 texforms was less consistent, the overall success with texforms highlights the importance of preserved low- and mid-level features.
  • Cross-Decoding Success: Decoders trained on one stimulus type (unaltered or texform) could generalize to classify the other, indicating shared underlying neural representations. However, performance was reduced compared to within-type decoding, suggesting that both low-level features (preserved in texforms) and higher-level structure (present in unaltered images) contribute.
  • Human-Neural Correlation: Monkey neural decoding performance was significantly correlated with human classification scores for texforms, particularly when decoding was performed along the "congruent" axis (e.g., decoding animacy from neural responses and correlating with human animacy scores).
  • No Temporal Difference: Decoding performance for both texforms and unaltered images did not differ significantly between transient and post-transient epochs, suggesting that early feedforward responses contribute substantially to this information.

Limitations and Future Work

  • Limited Neuron Count: The study decoded information from populations of "a few dozen neurons," which is a small fraction of the total neurons in V1 and V4. Future work could explore larger populations to see if decoding accuracy improves.
  • Linear Decoding: The study relied on linear decoders. It's possible that non-linear relationships exist that are not captured by this method.
  • Species Differences: While rhesus monkeys are excellent models for primate vision, there might be species-specific differences in visual processing.
  • Future Directions: The authors suggest investigating whether certain categories are more linearly decodable early on than others, potentially based on their ecological importance or the distinctiveness of their low-level features. They also highlight the value of merging neural recordings with human psychophysics to understand fundamental properties of neural coding.

Summary updated

Abstract

Abstract The complexity of visual features for which neurons are tuned increases from early to late stages of the ventral visual stream. Thus, the standard hypothesis is that high-level functions like object categorization are primarily mediated by higher visual areas because they require more complex image formats that are not evident in early visual processing stages. However, human observers can categorize images as objects or animals or as big or small even when the images preserve only some low- and mid-level features but are rendered unidentifiable (‘texforms’, Long et al., 2018). This observation suggests that even the early visual cortex, in which neurons respond to simple stimulus features, may already encode signals about these more abstract high-level categorical distinctions. We tested this hypothesis by recording from populations of neurons in early and mid-level visual cortical areas while rhesus monkeys viewed texforms and their unaltered source stimuli (simultaneous recordings from areas V1 and V4 in one animal and separate recordings from V1 and V4 in two others). Using recordings from a few dozen neurons, we could decode the real-world size and animacy of both unaltered images and texforms. Furthermore, this neural decoding accuracy across stimuli was related to the ability of human observers to categorize texforms by real-world size and animacy. Our results demonstrate that neuronal populations early in the visual hierarchy contain signals useful for higher-level object perception and suggest that the responses of early visual areas to simple stimulus features display preliminary untangling of higher-level distinctions.
BibTeX
@misc{kramer2023contributions,
  author = {Lily E. Kramer and Yi-Chia Chen and Bria Long and Talia Konkle and Marlene R. Cohen},
  title = {{Contributions of early and mid-level visual cortex to high-level object categorization}},
  year = {2023},
  doi = {10.1101/2023.05.31.541514},
  howpublished = {bioRxiv},
}
Figure