Are Convolutional Neural Networks or Transformers more like human vision?

Shikhar Tuli, Ishita Dasgupta, Erin Grant, Thomas L. Griffiths

TL;DR

This paper investigates whether Vision Transformers (ViTs) or traditional Convolutional Neural Networks (CNNs) better replicate human visual processing. By analyzing error patterns and "shape bias," the researchers find that ViTs are significantly more consistent with human vision than CNNs, suggesting that flexible attention mechanisms might be a better proxy for human-like recognition than hard-coded spatial constraints.

What Problem Does This Paper Address?

While modern AI models often match or exceed human accuracy on benchmarks like ImageNet, they frequently do so using fundamentally different strategies. A well-known discrepancy is the texture bias: CNNs tend to classify objects based on local textures (e.g., the skin of an elephant), whereas humans rely on global shapes.

The authors seek to determine if the newer Vision Transformer (ViT) architecture—which replaces the rigid local connectivity of convolutions with flexible self-attention—exhibits more "human-like" behavior. They go beyond simple accuracy metrics, focusing instead on error consistency: if a model makes the same mistakes as a human on the same images, it likely shares a similar underlying decision-making strategy.

Methods

The researchers compared a suite of standard CNNs (like ResNet-50, AlexNet, and VGG-16) against ViT models. Their approach involved:

  1. Error Consistency Metrics: They used Cohen’s $\kappa$, a standard metric that measures how often two systems agree on which images are misclassified, corrected for chance.
  2. Granular JS Distance: They introduced a new metric based on the Jensen-Shannon (JS) distance between error distributions. This allows for a more nuanced look at what a model confuses (e.g., misclassifying a car as a truck vs. a dog).
  3. Class-wise JS distance: Compares which specific classes are frequently missed.
  4. Inter-class JS distance: Compares the full "confusion matrix" to see if models and humans make similar types of misidentifications.
  5. Shape Bias Analysis: Using "cue-conflict" images where the shape of one object is filled with the texture of another, they measured how often models prioritized shape over texture.
  6. Fine-tuning: They tested whether training models on heavily augmented data (rotation, blur, color distortion) could bridge the gap between AI and human vision.
Fig. 2: Bird’s eye view of (a) convolutional and (b) attention- based networks.

Datasets

  • ImageNet: Used for initial large-scale pre-training.
  • Stylized ImageNet (SIN): A diagnostic dataset where textures from paintings are applied to ImageNet objects. This is the primary tool for measuring shape vs. texture bias.
  • WordNet Hierarchy: Used to cluster the 1,000 ImageNet classes into 16 "entry-level" categories (e.g., dog, car, bird) to make human-model comparisons feasible.

Key Insights and Ideas

  • ViTs are naturally more "human-like": Even without special training, ViTs showed higher error consistency with humans than any of the tested CNNs. This suggests that the "inductive bias" of CNNs (local spatial structure) might actually be a hindrance to human-like global recognition.
  • Shape Bias is the differentiator: ViTs exhibit a significantly higher shape bias than CNNs. This aligns them more closely with human developmental psychology, where shape is the primary cue for object categorization.
  • The "What" vs. "Which" Paradox: While ViTs are more consistent in which images they miss (class-wise), ResNets were surprisingly better at matching the specific types of errors humans make (inter-class). For example, a ResNet might be more likely to confuse a cat with a dog (a human-like error) than a ViT, even if the ViT is overall more accurate.
  • Augmentation is a double-edged sword: Fine-tuning on augmented data increased shape bias for both architectures, but it actually made ResNet less human-like in its overall error patterns.

Key Results

  • Error Consistency: ViT-B/32 achieved the highest Cohen’s $\kappa$ and lowest class-wise JS distance compared to human data, outperforming the ResNet-50 baseline.
  • Shape Bias: As shown in the diagnostic tests, ViTs consistently prioritized shape over texture more often than CNNs.
  • Accuracy Stability: When fine-tuned on augmented data to increase shape bias, ViTs maintained their ImageNet accuracy much better than ResNets, which saw a significant performance drop.
Fig. 3: (a) Class-wise and (b) inter-class JS distance vs. Cohen’s κ on the Stylized ImageNet (SIN) dataset. Class-wise JS distance vs. (c) ImageNet top-1 errors and (d) SIN top-1 error.

Limitations and Future Work

  • Human Confusion Matrices: The study notes the difficulty of collecting enough human data to populate a full 1000x1000 confusion matrix, necessitating the use of 16 broader categories.
  • Architecture Breadth: The study focused on standard ViTs and ResNets. Future work could explore hybrid architectures or different Transformer variants (like iGPT) to see if specific architectural "features" further enhance human-likeness.
  • Training Objectives: The authors suggest that human error-consistency metrics could be integrated directly into the loss function during training to force models toward more human-like strategies.

Summary updated

Abstract

Modern machine learning models for computer vision exceed humans in accuracy on specific visual recognition tasks, notably on datasets like ImageNet. However, high accuracy can be achieved in many ways. The particular decision function found by a machine learning system is determined not only by the data to which the system is exposed, but also the inductive biases of the model, which are typically harder to characterize. In this work, we follow a recent trend of in-depth behavioral analyses of neural network models that go beyond accuracy as an evaluation metric by looking at patterns of errors. Our focus is on comparing a suite of standard Convolutional Neural Networks (CNNs) and a recently-proposed attention-based network, the Vision Transformer (ViT), which relaxes the translation-invariance constraint of CNNs and therefore represents a model with a weaker set of inductive biases. Attention-based networks have previously been shown to achieve higher accuracy than CNNs on vision tasks, and we demonstrate, using new metrics for examining error consistency with more granularity, that their errors are also more consistent with those of humans. These results have implications both for building more human-like vision models, as well as for understanding visual object recognition in humans.
BibTeX
@misc{tuli2021are,
  author = {Shikhar Tuli and Ishita Dasgupta and Erin Grant and Thomas L. Griffiths},
  title = {{Are Convolutional Neural Networks or Transformers more like human vision?}},
  year = {2021},
  eprint = {2105.07197},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.07197},
}
Figure