Vision CNNs trained to estimate spatial latents learned similar ventral-stream-aligned representations

Yudi Xie, Weichen Huang, Esther Alter, Jeremy Schwartz, Joshua B. Tenenbaum, James J. DiCarlo

TL;DR

Training neural networks on simple spatial tasks—like estimating an object's distance or pose—produces internal representations that are remarkably similar to those found in the primate ventral visual stream. These "where" models achieve neural alignment scores comparable to models trained on massive "what" (categorization) datasets, suggesting that the brain's object recognition pathway may be optimized for a much broader set of visual latents than previously thought.

What Problem Does This Paper Address?

In neuroscience, the visual system is traditionally divided into two streams: the ventral stream (the "what" pathway) for object recognition and the dorsal stream (the "where" pathway) for spatial localization and action. Consequently, most AI models of the ventral stream are trained on object categorization tasks like ImageNet.

However, biological evidence shows the ventral stream also encodes rich spatial information, such as position and pose. This paper challenges the "categorization-first" dogma by asking: Could the ventral stream's functional role be better understood as estimating spatial latents? And if we train models on spatial tasks instead of categories, how different do their internal "brains" actually look?

Methods

The researchers trained standard Convolutional Neural Networks (ResNet-50 and ResNet-18) on various supervised objectives:

  • Spatial Latent Regression: Estimating continuous variables like distance ($Z$), translation ($X, Y$), and rotation (Euler angles).
  • Categorization: Classifying object categories (117 classes) or specific object identities (548 classes).
  • Multi-task Learning: Combinations of the above.

To evaluate these models, they used:

  1. Brain-Score: A standard benchmark that measures how well a model's internal activations can predict neural responses in primate brain regions (V1, V2, V4, and IT).
  2. Centered Kernel Alignment (CKA): A metric to quantify the similarity between the internal representations of different models.
  3. Linear Decoding: Probing layers to see if they "inadvertently" learned information they weren't explicitly trained on.

Datasets

The study primarily utilized a new synthetic dataset, TDW-117, generated using the ThreeDWorld (TDW) physics engine.

Figure 1: Spatial latent variables and our training dataset. (a) An illustration of the set of spatial latents that are available in the dataset for training models. In addition to the object category (Apple), the set of spatial latent variables we record are the following: translation (X, Y), distance (Z), and rotation (Rxy, Ryz, Rzx). (This image is for illustration only, not in the synthetic dataset.) (b) Example images in our dataset (TDW-117) for training CNNs. Each image contains one object with varying positions and poses against a random background.
  • TDW-117: Contains 1.3 million to 100 million images of 548 object models across 117 categories, rendered with random positions, poses, and backgrounds.
  • ImageNet-1K: Used as a natural-image baseline for categorization training.
  • TDW-N: Controlled subsets of the synthetic data with varying numbers of categories to test how task complexity affects neural alignment.

Key Insights and Ideas

  • Spatial Tasks are Sufficient: You don't need to learn 1,000 categories to develop a brain-like visual system. A model trained only to estimate object distance (a single output unit) achieves neural alignment nearly identical to one trained on hundreds of categories.
  • Representational Convergence: Despite having vastly different goals, "what" and "where" models develop highly similar features in their early and middle layers. They only truly diverge at the very end of the network.
  • The "Free Lunch" of Data Variability: Why do these models converge? The authors hypothesize that if a dataset contains variability in factors you aren't training on (e.g., different objects appearing at different distances), the model learns to represent those "non-target" factors anyway to solve its primary task.
  • Ventral Stream as a Generalist: These results suggest the ventral stream might not be a specialized "categorization engine," but rather a provider of general-purpose visual representations that support a wide array of downstream tasks.

Key Results

The study found that models trained on just a few spatial latents reached surprisingly high neural alignment scores.

Figure 2: Learning a small number of spatial latents produces ventral-stream-aligned CNN models. (a) The neural alignment of models (ResNet-50) trained on different objectives (x-axis, number of output units, see panel b). Learning a few spatial latent variables produced models that have neural alignment scores comparable to models trained on hundreds of categories. Error bars or shaded regions show the SD across multiple random seeds (N=5). All spatial + classification = all spatial and all classification tasks combined. TDW-117 is our TDW dataset with 117 object cat- egories; TDW-N means datasets with N = 2,4,6,8,16 categories. For a breakdown of the individual region alignment scores, see Figure C.1. (b) The training tasks we investigated and their correspond- ing number of output units that receive supervision during training.
  • Neural Alignment: A model trained only on Distance (1 output unit) achieved a mean neural score of 0.399, which is 97% of the score achieved by a model trained on all 117 categories (0.412). For context, an ImageNet-trained model scores 0.430.
  • Performance Correlation: There is a near-perfect correlation (Pearson $r = 0.97$) between how well a model performs its spatial task and how well it aligns with the brain.
Figure 3: Strong correlations between task performance (negative loss) and neural alignment across different spatial objectives.
  • Variability Drives Learning: Causal experiments showed that adding category variability to a distance-training task significantly improved the model's ability to decode categories from its middle layers, even though it was never told what a "category" was.
Figure 5: Experimental evidence showing that non-target latent variability (like having many categories) helps models learn better internal representations of those latents.

Limitations and Future Work

  • The Synthetic Gap: While synthetic-trained models are highly brain-like, they still don't quite reach the performance of models trained on natural images (ImageNet). Improving rendering quality and scene diversity is a clear next step.
  • Behavioral Mismatch: Interestingly, while spatial models align well with neural data, they don't align as well with human behavior in categorization tasks. This suggests that while the features are there, the "readout" for behavior might be more specialized.
  • IT Benchmark Variance: On some specific high-level (IT) benchmarks, spatial models still underperform category-trained ones, indicating that some aspects of high-level vision might still require explicit categorization training.

Summary updated

Abstract

Studies of the functional role of the primate ventral visual stream have traditionally focused on object categorization, often ignoring -- despite much prior evidence -- its role in estimating "spatial" latents such as object position and pose. Most leading ventral stream models are derived by optimizing networks for object categorization, which seems to imply that the ventral stream is also derived under such an objective. Here, we explore an alternative hypothesis: Might the ventral stream be optimized for estimating spatial latents? And a closely related question: How different -- if at all -- are representations learned from spatial latent estimation compared to categorization? To ask these questions, we leveraged synthetic image datasets generated by a 3D graphic engine and trained convolutional neural networks (CNNs) to estimate different combinations of spatial and category latents. We found that models trained to estimate just a few spatial latents achieve neural alignment scores comparable to those trained on hundreds of categories, and the spatial latent performance of models strongly correlates with their neural alignment. Spatial latent and category-trained models have very similar -- but not identical -- internal representations, especially in their early and middle layers. We provide evidence that this convergence is partly driven by non-target latent variability in the training data, which facilitates the implicit learning of representations of those non-target latents. Taken together, these results suggest that many training objectives, such as spatial latents, can lead to similar models aligned neurally with the ventral stream. Thus, one should not assume that the ventral stream is optimized for object categorization only. As a field, we need to continue to sharpen our measures of comparing models to brains to better understand the functional roles of the ventral stream.
BibTeX
@inproceedings{xie2025vision,
  author = {Yudi Xie and Weichen Huang and Esther Alter and Jeremy Schwartz and Joshua B. Tenenbaum and James J. DiCarlo},
  title = {{Vision CNNs trained to estimate spatial latents learned similar ventral-stream-aligned representations}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=emMMa4q0qw},
}
Figure