Representations of regular and irregular shapes by deep Convolutional Neural Networks, monkey inferotemporal neurons and human judgments

Vogels Rufin, Ioannis Kalfas, Kasper Vinken, Rufin VogelsPLOS Computational Biology 2018 · DOI

TL;DR

Deep Convolutional Neural Networks (CNNs) trained on natural images show representations of abstract shapes that closely mirror those found in the inferotemporal (IT) cortex of macaque monkeys. Interestingly, while IT neurons' representations align with earlier CNN layers, human judgments of shape similarity are better captured by the deepest layers of these networks.

What Problem Does This Paper Address?

The primate inferotemporal (IT) cortex is a crucial brain region for object recognition. While studies have shown that deep CNNs trained on natural images can model IT cortex responses to those same natural images, it's less clear if these models can generalize to simpler, abstract shapes that have historically been used to probe IT neuron selectivity. This paper investigates whether deep CNNs, trained on complex natural scenes, can capture the way IT neurons represent and differentiate between regular and irregular 2D shapes, and how this compares to human perception.

Methods

The researchers employed representational similarity analysis (RSA) to compare the patterns of neural activity and network activations. They used three state-of-the-art deep CNN architectures (Alexnet, VGG-16, and VGG-19) in both their trained (on ImageNet) and untrained (random weights) versions. For each layer of these CNNs, they extracted activation patterns in response to a set of 64 abstract shapes. These patterns were then used to compute dissimilarity matrices, which were compared to dissimilarity matrices derived from macaque IT neuron recordings and human similarity judgments of the same shapes. They used both Euclidean distance and Spearman rank correlation as metrics for dissimilarity.

Datasets

  • Stimuli: A set of 64 abstract 2D shapes, comprising regular shapes and three categories of irregular shapes (Irregular Complex, Irregular Simple Curved, Irregular Simple Straight). This set was previously used by Kayaert et al. [12].
  • Training Data for CNNs: ImageNet dataset (used for pre-training the CNN models).

Key Insights and Ideas

  • CNNs as IT Models: Trained deep CNNs, even though not explicitly trained on abstract shapes, develop representations that are remarkably similar to those of macaque IT neurons, particularly in their deeper convolutional layers.
  • The Role of Training: Untrained CNNs (with random weights) showed significantly poorer similarity to IT neuron responses compared to trained CNNs, highlighting the crucial role of the classification training process in shaping these representations.
  • Layer-Specific Similarities: The similarity between IT neurons and CNNs generally increased with the depth of the CNN layers, peaking in the deep convolutional layers. However, this trend reversed for the fully connected layers in some networks.
  • Human vs. Monkey Perception: While IT neurons' representations aligned well with earlier CNN layers, human similarity judgments showed a stronger correlation with the deepest layers of the trained CNNs.
  • Sensitivity to Non-Accidental Properties (NAPs): Deep CNNs, like humans, showed an enhanced sensitivity to differences in non-accidental properties (e.g., straight vs. curved contours) in irregular shapes, a bias that emerged through training.

Key Results

  • The representational similarity between macaque IT neurons and deep convolutional layers of trained CNNs reached a peak correlation around 0.60-0.64.
  • Untrained CNNs showed a much weaker similarity to IT neurons, suggesting that classification training is essential for capturing these neural representations.
  • Human similarity judgments correlated best with the deepest layers of the trained CNNs, particularly showing a bias for straight vs. curved irregular shapes, mirroring findings from human behavioral studies.
  • The study found that the representations of abstract shape similarity are highly comparable between macaque IT neurons and deep convolutional layers of CNNs trained on natural images.

Limitations and Future Work

  • Generalizability: The authors acknowledge that while the findings are compelling, the high similarity might not generalize to all possible shape properties not varied in this study.
  • Mapping CNN Layers to Brain Areas: The exact neural correlates of different CNN layers remain an open question. While shallow layers might relate to early visual areas and deeper layers to IT, the precise mapping is complex and may involve mixed representations.
  • Model Improvement: Despite CNNs being the best models to date for primate object recognition, they still have limitations, such as susceptibility to noisy stimuli or adversarial examples. Future work could explore recurrent connectivity or other architectural improvements.
  • Future Research: The authors suggest further research involving recordings from different laminae of various visual areas and mapping them to CNN units to better understand the neural basis of CNN representations. They also emphasize the value of using untrained CNNs as controls in future studies.

Summary updated

BibTeX
@article{rufin2018representations,
  author = {Vogels Rufin and Ioannis Kalfas and Kasper Vinken and Rufin Vogels},
  title = {{Representations of regular and irregular shapes by deep Convolutional Neural Networks, monkey inferotemporal neurons and human judgments}},
  journal = {PLOS Computational Biology},
  year = {2018},
  doi = {10.1371/journal.pcbi.1006557},
}
Figure