Abstract
Object classification has been proposed as a principal objective of the primate ventral visual stream and has been used as an optimization target for deep neural network models (DNNs) of the visual system. However, visual brain areas represent many different types of information, and optimizing for classification of object identity alone does not constrain how other information may be encoded in visual representations. Information about different scene parameters may be discarded altogether (“invariance”), represented in non-interfering subspaces of population activity (“factorization”) or encoded in an entangled fashion. In this work, we provide evidence that factorization is a normative principle of biological visual representations. In the monkey ventral visual hierarchy, we found that factorization of object pose and background information from object identity increased in higher-level regions and strongly contributed to improving object identity decoding performance. We then conducted a large-scale analysis of factorization of individual scene parameters – lighting, background, camera viewpoint, and object pose – in a diverse library of DNN models of the visual system. Models which best matched neural, fMRI and behavioral data from both monkeys and humans across 12 datasets tended to be those which factorized scene parameters most strongly. Notably, invariance to these parameters was not consistently associated with matches to neural and behavioral data, suggesting that maintaining non-class information in factorized activity subspaces is preferred to dropping it altogether. Thus, we propose that factorization of visual scene information is a widely used strategy in brains and DNN models thereof.
Introduction
For the past decade, the dominant paradigm for modeling the primate ventral stream—the "what" pathway—has been training deep neural networks (DNNs) on object classification. While these models are remarkably good at predicting neural responses, researchers have hit a "classification plateau": beyond a certain point, making a model better at classification doesn't make it more brain-like.
This paper, published in eLife, suggests that this plateau exists because the brain isn't just a classifier; it’s a scene-understanding engine. While a classifier might discard information about an object's rotation or the background to simplify its task (invariance), the brain needs that data for navigation and manipulation. The challenge is representing this "nuisance" information without "entangling" it and making object recognition harder.
The Principle of Factorization
The authors argue for factorization, a strategy where the brain preserves non-essential information but organizes it into independent, non-interfering subspaces. This contrasts with invariance, which collapses these variables to focus solely on identity.
Factorization is a "win-win" strategy. Unlike invariance, which is zero-sum (to be more invariant to pose, you must lose information about pose), factorization allows for high-performance object decoding while preserving high-dimensional scene information.
Methods and Datasets
The researchers introduced a formal metric to distinguish between invariance (reducing response variance to a parameter) and factorization (ensuring the variance driven by one parameter is orthogonal to the variance driven by others). They tested these metrics across 12 datasets, including macaque neural recordings (V4 and IT), human fMRI, and behavioral data.
To benchmark DNNs, they used a custom synthetic image set where scene parameters—lighting, background, camera viewpoint, and object pose—were precisely controlled.
Key Results
- Factorization Increases Up the Hierarchy: As visual information moves from V4 to IT, it becomes significantly more factorized. IT is better at separating identity from pose and background than lower-level regions.
- Predicting Brain-Likeness: The degree to which a DNN factorizes scene information is a much stronger predictor of its "neural predictivity" than its classification accuracy alone. Factorization scores for object pose and camera viewpoint were highly correlated with a model's ability to fit neural and fMRI data (r ≈ 0.6 to 0.8).
- Invariance Can Be Misleading: Interestingly, invariance to spatial transforms often had a negative or zero correlation with brain-likeness, suggesting that "dropping" this information is un-biological.
Conclusion
Factorization appears to be a normative principle of biological visual representations. By maintaining non-class information in factorized activity subspaces rather than discarding it, the primate visual system achieves both robust object recognition and detailed scene understanding. Future work will need to identify the specific neural mechanisms—whether physical subpopulations or distributed high-dimensional subspaces—that enable this elegant organizational strategy.
Summary updated