Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs

Jonas Kubilius, Martin Schrimpf, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel L. K. Yamins, James J. DiCarlo

TL;DR

Researchers have developed CORnet-S, a shallow, recurrent artificial neural network that matches the primate brain's visual processing architecture more closely than massive deep learning models. Despite having significantly fewer layers, CORnet-S achieves state-of-the-art scores on "Brain-Score" benchmarks and remains highly competitive on ImageNet, proving that biological plausibility and engineering performance can go hand-in-hand.

The Disconnect Between AI and Neuroscience

In recent years, the fields of AI and neuroscience have drifted apart. While early Artificial Neural Networks (ANNs) like AlexNet were inspired by the brain's anatomy, modern state-of-the-art models (like ResNet or NASNet) have become "black boxes" with hundreds of layers and complex branching structures.

This architectural divergence creates two major problems:

  1. Anatomical Mismatch: The primate ventral stream (the "what" pathway for object recognition) consists of only a few distinct stages (V1, V2, V4, and IT). Mapping a 152-layer ResNet onto these four areas is nearly impossible.
  2. Missing Dynamics: The brain is highly recurrent—information flows back and forth within and between areas. Most high-performing AI models are purely feed-forward, meaning they cannot simulate the temporal "evolution" of a thought or a recognition event.

The authors ask a fundamental question: Can we build a model that is compact and biologically mapped, yet still performs as well as the deep "monsters" of machine learning?

Methods: Introducing CORnet-S

To bridge this gap, the authors introduced CORnet-S (the "S" stands for Shallow). Its design is guided by three core principles:

  • Anatomical Mapping: The model is strictly divided into four areas corresponding to the primate visual system: V1, V2, V4, and IT.
  • Recurrence: Unlike standard feed-forward nets, CORnet-S uses within-area recurrence. Information passes through the same circuitry multiple times (unrolled over time), allowing the model to "think" longer on difficult images without adding new physical layers.
  • Compactness: By using recurrence and weight sharing, the model maintains a "shallow" feed-forward depth of only 15 layers, compared to the 50+ layers common in high-performing models.
CORnet-S architecture mapping brain areas to a recurrent ANN and its leading Brain-Score performance.

To evaluate "brain-likeness," the researchers utilized Brain-Score, a large-scale composite benchmark that compares model activations to actual neural firing rates in monkeys, as well as human behavioral patterns (e.g., which images humans find easy or hard to recognize).

Datasets and Generalization

The model was primarily trained and benchmarked for object recognition on ImageNet 2012. However, to truly test its biological fidelity, the authors evaluated CORnet-S across several generalization sets:

  • Brain-Score: A suite of neural recordings from primate areas V4 and IT, along with human behavioral data.
  • Generalization Sets: The authors tested the model on new neural data (monkeys viewing MS COCO images), new behavioral data (Amazon Mechanical Turk), and transfer learning tasks (CIFAR-100).
Generalization performance of CORnet-S (red) compared to other models (green) across new neural, behavioral, and transfer learning datasets.

Key Insights and Results

  • Performance Does Not Equal Brain-Likeness: One of the most striking findings is that the highest-performing models on ImageNet (like PNASNet) are often less brain-like. Simply chasing higher accuracy leads to architectures that diverge from biology.
  • Recurrence is Key: Through extensive circuitry analysis, the authors found that recurrence was the single most important factor for improving both Brain-Score and ImageNet performance in shallow models.
  • #1 on Brain-Score: CORnet-S achieved a Brain-Score of 0.471, outperforming much deeper models like ResNet-101 and DenseNet.
  • Competitive ImageNet Performance: It reached 73.1% top-1 accuracy. While lower than the absolute state-of-the-art, it is the best-performing model among "shallow" architectures (depth < 20).
  • Capturing "Object Solution Time" (OST): Because CORnet-S is recurrent, it has a temporal dimension. The authors showed that the time it takes for the model to "settle" on an answer correlates with the time it takes for a monkey's brain to resolve the same object (achieving a 0.25 correlation with monkey IT population dynamics—a benchmark where feed-forward models automatically score zero).

Limitations and Future Work

While CORnet-S is a significant step forward, it remains a simplified abstraction:

  • Missing Pathways: It currently lacks across-area feedback (e.g., IT sending signals back to V4) and does not explicitly model the retina or the Lateral Geniculate Nucleus (LGN).
  • Learning Rules: The model is still trained using backpropagation, which is not considered biologically plausible.
  • Temporal Resolution: While it captures the trend of neural dynamics, a correlation of 0.25 suggests there is still significant room to improve how models simulate the brain's precise timing.

Future iterations of Brain-Score and CORnet will likely incorporate more diverse circuitry and more stringent biological constraints to further bridge the gap between artificial intelligence and neuroscience.

Summary updated

Abstract

Deep convolutional artificial neural networks (ANNs) are the leading class of candidate models of the mechanisms of visual processing in the primate ventral stream. While initially inspired by brain anatomy, over the past years, these ANNs have evolved from a simple eight-layer architecture in AlexNet to extremely deep and branching architectures, demonstrating increasingly better object categorization performance, yet bringing into question how brain-like they still are. In particular, typical deep models from the machine learning community are often hard to map onto the brain's anatomy due to their vast number of layers and missing biologically-important connections, such as recurrence. Here we demonstrate that better anatomical alignment to the brain and high performance on machine learning as well as neuroscience measures do not have to be in contradiction. We developed CORnet-S, a shallow ANN with four anatomically mapped areas and recurrent connectivity, guided by Brain-Score, a new large-scale composite of neural and behavioral benchmarks for quantifying the functional fidelity of models of the primate ventral visual stream. Despite being significantly shallower than most models, CORnet-S is the top model on Brain-Score and outperforms similarly compact models on ImageNet. Moreover, our extensive analyses of CORnet-S circuitry variants reveal that recurrence is the main predictive factor of both Brain-Score and ImageNet top-1 performance. Finally, we report that the temporal evolution of the CORnet-S "IT" neural population resembles the actual monkey IT population dynamics. Taken together, these results establish CORnet-S, a compact, recurrent ANN, as the current best model of the primate ventral visual stream.
BibTeX
@inproceedings{kubilius2019brain,
  author = {Jonas Kubilius and Martin Schrimpf and Ha Hong and Najib J. Majaj and Rishi Rajalingham and Elias B. Issa and Kohitij Kar and Pouya Bashivan and Jonathan Prescott-Roy and Kailyn Schmidt and Aran Nayebi and Daniel Bear and Daniel L. K. Yamins and James J. DiCarlo},
  title = {{Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure