TL;DR
This paper introduces CORnet-S, a shallow, recurrent neural network that achieves state-of-the-art performance on a new benchmark for brain-likeness (Brain-Score) while maintaining competitive accuracy on ImageNet. This demonstrates that models can be both anatomically aligned with the brain and highly performant, challenging the trend of increasingly deep and complex architectures.
What Problem Does This Paper Address?
Deep Convolutional Neural Networks (CNNs) have become the leading computational models for understanding visual processing in the primate brain. While initially inspired by brain anatomy, modern CNNs have evolved into extremely deep and complex architectures, often with hundreds of layers. This depth makes it difficult to map them onto the relatively shallow structure of the primate ventral visual stream. Furthermore, these deep models often lack biologically important features like recurrence, which recent studies suggest plays a crucial role in visual processing. The paper addresses the question of whether it's possible to build models that are both anatomically constrained (shallow, recurrent) and achieve high performance on both machine learning tasks and neuroscience-based benchmarks.
Methods
The core contribution is the development of CORnet-S, a novel neural network architecture. Key methodological aspects include:
- Anatomical Alignment: CORnet-S is designed with a shallow, four-layer structure, conceptually mapped to primate visual areas V1, V2, V4, and IT. This contrasts with the hundreds of layers in typical state-of-the-art CNNs.
- Recurrent Connectivity: Unlike most feedforward CNNs, CORnet-S incorporates recurrent connections within its V2, V4, and IT areas. This allows the network to process information over time, mimicking temporal dynamics observed in the brain.
- Brain-Score Benchmark: The authors introduce and utilize Brain-Score, a new composite benchmark designed to quantify the functional fidelity of models to the primate ventral visual stream. It combines neural recordings and behavioral data from primates and humans.
- Circuitry Analysis: The paper includes an analysis of CORnet-S variants to identify which architectural components (e.g., recurrence, bottleneck size, skip connections) are most critical for its performance on Brain-Score and ImageNet.
Datasets
- ImageNet 2012: Used for training CORnet-S and evaluating its object categorization performance.
- Primate Neural Recordings: Data from V4 and IT areas of non-human primates used for the neural predictivity component of Brain-Score.
- Human Behavioral Data: Data from human object recognition tasks used for the behavioral predictivity component of Brain-Score.
- MS COCO: Used for evaluating generalization to new datasets.
- CIFAR-100: Used for evaluating transfer learning performance.
Key Insights and Ideas
- Anatomical Constraints Can Enhance Brain-Likeness: The paper strongly suggests that adhering to anatomical constraints (shallowness, recurrence) doesn't necessarily sacrifice performance; in fact, it can lead to models that are more brain-like.
- Recurrence is Key for Brain-Like Performance: The analysis highlights recurrence as the most significant factor driving CORnet-S's high scores on both Brain-Score and ImageNet. This reinforces the importance of temporal dynamics in visual processing.
- Brain-Score as a Unifying Metric: The Brain-Score benchmark proves effective in evaluating models beyond simple accuracy, revealing a potential disconnect between ImageNet performance and brain fidelity.
- Shallow Models Can Be Competitive: CORnet-S demonstrates that highly performant models don't need to be excessively deep, challenging the "bigger is better" paradigm in deep learning for neuroscience modeling.
Key Results
- Top Performer on Brain-Score: CORnet-S achieved the highest score on the Brain-Score benchmark, outperforming a wide range of popular deep neural networks.
- Competitive ImageNet Performance: CORnet-S achieved a top-1 accuracy of 73.1% on ImageNet, competitive with similarly compact models.
- Recurrence's Impact: Analysis showed that recurrence was the primary driver for both Brain-Score and ImageNet top-1 performance improvements in CORnet-S variants.
- Captures Neural Dynamics: CORnet-S was able to capture the temporal evolution of neural activity in the primate IT cortex, a feat impossible for feedforward models. It achieved a correlation of 0.25 with monkey "object solution times" (OSTs).
- Shallow Architecture: CORnet-S has a depth of 15 layers, significantly shallower than many state-of-the-art models, yet it outperforms other shallow models on ImageNet and CIFAR-100 transfer tasks.
Limitations and Future Work
- Learning Mechanisms: The authors acknowledge that CORnet-S, like most current ANNs, lacks biologically plausible learning mechanisms (e.g., self-supervised or unsupervised learning).
- Anatomical Completeness: While CORnet-S is more anatomically aligned than many models, it still omits details like the retina or lateral geniculate nucleus.
- Generalizability of Brain-Score: The authors suggest that adding more benchmarks to Brain-Score could further refine its ability to evaluate models.
- Hyperparameter Optimization: The authors note that while they tested many architectures, they cannot rule out that optimal hyperparameters for some variants might minimize reported differences.
- Future Directions: The paper opens avenues for exploring less constrained anatomical mappings and more diverse circuitry, as well as investigating the impact of different training strategies on brain-like performance.
Summary updated