High-dimensional topographic organization of visual features in the primate temporal lobe

Mengna Yao, Bincheng Wen, Mingpo Yang, Jiebin Guo, Haozhou Jiang, Chao Feng, Yilei Cao, Huiguang He, Le ChangNature Communications 2023 · OpenReview

Abstract

The inferotemporal cortex (ITC) is central to primate object recognition, yet its global functional organization remains partially understood. This study utilizes deep neural networks (DNNs) to construct a 25-dimensional space of visual features, mapping the spatial organization of feature preferences in both macaque and human brains using high-resolution fMRI. The resulting maps reveal a pair of orthogonal gradients that vary in spatial scale and highlight significant differences in functional organization between species. These findings provide a unified framework for understanding the topographic layout of high-level visual areas.

Introduction

The inferotemporal cortex (ITC) represents the pinnacle of the ventral visual stream, enabling the sophisticated object recognition capabilities of primates. While previous research has identified specialized "patches" for categories like faces, bodies, and places, these regions account for only about half of the monkey ITC. A comprehensive theory explaining the global organization of the temporal lobe—whether it is a collection of discrete modules or a continuous feature map—has remained elusive. Furthermore, the extent to which human and non-human primate "hardware" layouts correspond is not fully understood. This research seeks to define the high-dimensional coordinates the brain uses to navigate the visual world, moving beyond simple categorical labels.

Methodology

The researchers adopted a "model-first" strategy to map the brain's functional architecture. First, they extracted features by passing 200,000 images from the ImageNet dataset through AlexNet, a standard convolutional neural network. By applying Independent Component Analysis (ICA) to the penultimate layer (fc7), they defined a 25-dimensional (25D) space of independent visual features.

The experimental pipeline: AlexNet's fc7 layer features are processed using ICA to establish a 25-dimensional visual feature space for brain mapping.

To map these features in the brain, fMRI was conducted on three macaque monkeys and four humans. Subjects were shown "positive" and "negative" images representing the extremes of each of the 25 feature axes to identify voxel-wise preferences. The resulting 25 maps were projected onto the cortical surface, allowing each ITC location to be treated as a 25D vector. Principal Component Analysis (PCA) was then used to identify the primary axes of spatial variation. Finally, Topographic Neural Networks (TNNs)—DNNs with spatial constraints—were used to model how such organization might emerge.

Key Insights and Results

The analysis revealed that the ITC's topographic map is highly complex, requiring at least 10 significant dimensions to describe accurately. This complexity far exceeds simpler 1D or 2D models based on animacy or size. A subset of seven interpretable features, such as spatial frequency and curvature, explained approximately 56% of neural variance, reaching 82.5% of the noise ceiling when supplemented with residual DNN features.

Each visual stimulus is mapped into a 25-dimensional space, where independent components (ICs) serve as the coordinate axes for neural preference.

A major discovery was the "Orthogonal Gradient Principle": the brain organizes information via two overlapping spatial gradients. A coarse dorsal-to-ventral gradient (PC1) is overlaid by a finer, periodic gradient (PC2) running parallel to the Superior Temporal Sulcus (STS). Interestingly, while both species share a similar 25D representation ($r \approx 0.83$), the spatial layout is remarkably "crystalline" and ordered in monkeys, whereas it appears significantly more fragmented and distorted in humans. This suggests that human brain evolution, driven by the need to process artificial objects like tools and text, may have reconfigured the ancestral primate map.

Discussion and Future Directions

The study provides a powerful new lens for viewing the ITC, identifying previously unknown functional regions like the "inanimate stripes" in macaques. However, limitations remain. The use of static 2D images means that 3D cues (disparity) and dynamic information (motion) were not captured in the 25D space. Additionally, the linear mapping between DNN features and neural responses, while effective, leaves a portion of variance unexplained. Future research utilizing self-supervised models or ecologically diverse training tasks may further bridge this gap, offering deeper insights into the non-linear complexities of primate vision.

Summary updated

Abstract

The inferotemporal cortex supports our supreme object recognition ability. Numerous studies have been conducted to elucidate the functional organization of this brain area, but there are still important questions that remain unanswered, including how this organization differs between humans and non-human primates. Here, we use deep neural networks trained on object categorization to construct a 25-dimensional space of visual features, and systematically measure the spatial organization of feature preference in both male monkey brains and human brains using fMRI. These feature maps allow us to predict the selectivity of a previously unknown region in monkey brains, which is corroborated by additional fMRI and electrophysiology experiments. These maps also enable quantitative analyses of the topographic organization of the temporal lobe, demonstrating the existence of a pair of orthogonal gradients that differ in spatial scale and revealing significant differences in the functional organization of high-level visual areas between monkey and human brains. The functional organisation of the inferotemporal cortex is not well understood. Here, the authors construct a multi-dimensional space of visual features using deep neural networks, and show the spatial organisation of feature preference in both human and monkey inferotemporal cortex.
BibTeX
@inproceedings{yao2023high,
  author = {Mengna Yao and Bincheng Wen and Mingpo Yang and Jiebin Guo and Haozhou Jiang and Chao Feng and Yilei Cao and Huiguang He and Le Chang},
  title = {{High-dimensional topographic organization of visual features in the primate temporal lobe}},
  booktitle = {Nature Communications},
  year = {2023},
  url = {https://openreview.net/forum?id=7cyI7WgAe0},
}
Figure