TL;DR
Training neural networks on simple spatial tasks—like estimating an object's distance or pose—produces internal representations that are remarkably similar to those found in the primate ventral visual stream. These "where" models achieve neural alignment scores comparable to models trained on massive "what" (categorization) datasets, suggesting that the brain's object recognition pathway may be optimized for a much broader set of visual latents than previously thought.
What Problem Does This Paper Address?
In neuroscience, the visual system is traditionally divided into two streams: the ventral stream (the "what" pathway) for object recognition and the dorsal stream (the "where" pathway) for spatial localization and action. Consequently, most AI models of the ventral stream are trained on object categorization tasks like ImageNet.
However, biological evidence shows the ventral stream also encodes rich spatial information, such as position and pose. This paper challenges the "categorization-first" dogma by asking: Could the ventral stream's functional role be better understood as estimating spatial latents? And if we train models on spatial tasks instead of categories, how different do their internal "brains" actually look?
Methods
The researchers trained standard Convolutional Neural Networks (ResNet-50 and ResNet-18) on various supervised objectives:
- Spatial Latent Regression: Estimating continuous variables like distance ($Z$), translation ($X, Y$), and rotation (Euler angles).
- Categorization: Classifying object categories (117 classes) or specific object identities (548 classes).
- Multi-task Learning: Combinations of the above.
To evaluate these models, they used:
- Brain-Score: A standard benchmark that measures how well a model's internal activations can predict neural responses in primate brain regions (V1, V2, V4, and IT).
- Centered Kernel Alignment (CKA): A metric to quantify the similarity between the internal representations of different models.
- Linear Decoding: Probing layers to see if they "inadvertently" learned information they weren't explicitly trained on.
Datasets
The study primarily utilized a new synthetic dataset, TDW-117, generated using the ThreeDWorld (TDW) physics engine.
- TDW-117: Contains 1.3 million to 100 million images of 548 object models across 117 categories, rendered with random positions, poses, and backgrounds.
- ImageNet-1K: Used as a natural-image baseline for categorization training.
- TDW-N: Controlled subsets of the synthetic data with varying numbers of categories to test how task complexity affects neural alignment.
Key Insights and Ideas
- Spatial Tasks are Sufficient: You don't need to learn 1,000 categories to develop a brain-like visual system. A model trained only to estimate object distance (a single output unit) achieves neural alignment nearly identical to one trained on hundreds of categories.
- Representational Convergence: Despite having vastly different goals, "what" and "where" models develop highly similar features in their early and middle layers. They only truly diverge at the very end of the network.
- The "Free Lunch" of Data Variability: Why do these models converge? The authors hypothesize that if a dataset contains variability in factors you aren't training on (e.g., different objects appearing at different distances), the model learns to represent those "non-target" factors anyway to solve its primary task.
- Ventral Stream as a Generalist: These results suggest the ventral stream might not be a specialized "categorization engine," but rather a provider of general-purpose visual representations that support a wide array of downstream tasks.
Key Results
The study found that models trained on just a few spatial latents reached surprisingly high neural alignment scores.
- Neural Alignment: A model trained only on Distance (1 output unit) achieved a mean neural score of 0.399, which is 97% of the score achieved by a model trained on all 117 categories (0.412). For context, an ImageNet-trained model scores 0.430.
- Performance Correlation: There is a near-perfect correlation (Pearson $r = 0.97$) between how well a model performs its spatial task and how well it aligns with the brain.
- Variability Drives Learning: Causal experiments showed that adding category variability to a distance-training task significantly improved the model's ability to decode categories from its middle layers, even though it was never told what a "category" was.
Limitations and Future Work
- The Synthetic Gap: While synthetic-trained models are highly brain-like, they still don't quite reach the performance of models trained on natural images (ImageNet). Improving rendering quality and scene diversity is a clear next step.
- Behavioral Mismatch: Interestingly, while spatial models align well with neural data, they don't align as well with human behavior in categorization tasks. This suggests that while the features are there, the "readout" for behavior might be more specialized.
- IT Benchmark Variance: On some specific high-level (IT) benchmarks, spatial models still underperform category-trained ones, indicating that some aspects of high-level vision might still require explicit categorization training.
Summary updated