TL;DR
While Convolutional Neural Networks (CNNs) often match human accuracy on image recognition, this paper reveals they use fundamentally different strategies. By measuring "error consistency"—whether two systems fail on the exact same images—the authors show that CNNs are remarkably consistent with each other but share almost no common error patterns with humans, even when using "brain-inspired" recurrent architectures.
What Problem Does This Paper Address?
In the quest to build "human-like" AI, researchers often use accuracy as the primary yardstick. However, accuracy is an aggregate (or "molar") metric; it tells you how often a system succeeds, but not how it succeeds. Two systems can achieve 90% accuracy using completely different logic—one might struggle with lighting, while the other struggles with textures.
As CNNs approach "ceiling performance" (near 100% accuracy) on standard datasets like ImageNet, accuracy becomes a blunt instrument for differentiation. This paper addresses the need for a "molecular" approach: a trial-by-trial analysis to determine if the underlying decision-making strategies of AI models actually resemble biological vision.
Methods
The authors introduce trial-by-trial error consistency as a quantitative tool. The core idea is to see if two observers (two models, two humans, or one of each) make the same errors on the same specific images.
- Observed Overlap ($c_{obs}$): The fraction of trials where both observers gave the same type of response (both correct or both incorrect).
- Expected Overlap ($c_{exp}$): The overlap you would expect by chance, given the individual accuracies of the two observers.
- Cohen’s Kappa ($\kappa$): To get a clean metric, the authors use $\kappa$ to normalize the data:
$$\kappa = \frac{c_{obs} - c_{exp}}{1 - c_{exp}}$$
- $\kappa = 1$: Perfect consistency.
- $\kappa = 0$: No more consistency than expected by chance (independent strategies).
- $\kappa < 0$: Systematic disagreement (inverse strategies).
The authors also provide analytical bounds and confidence intervals to ensure that observed consistencies aren't just statistical noise.
Datasets
The study evaluates 16 standard ImageNet-trained CNNs (AlexNet, ResNets, VGGs, etc.) and human observers across four types of stimuli:
- Standard ImageNet: High-quality color photos.
- Cue Conflict: Images where texture and shape point to different objects (e.g., a cat made of elephant skin), created via neural style transfer.
- Edge Stimuli: Simplified line drawings.
- Silhouettes: Blacked-out object shapes.
Key Insights and Ideas
- CNNs are a "Monoculture": Regardless of architecture (whether it's an old-school AlexNet or a modern DenseNet), CNNs are remarkably consistent with one another. They tend to find the same images "easy" or "hard," suggesting they all rely on similar statistical shortcuts.
- The Human-AI Gap is Wide: Human-to-human error consistency is moderate, but CNN-to-human consistency is strikingly low—often near zero for edge drawings and cue-conflict images. This implies that even when a CNN gets an answer right, it’s likely not "seeing" the image the way a human does.
- Recurrence isn't a Silver Bullet: The paper highlights CORnet-S, a recurrent model often cited as the "best model of the primate visual stream" based on its ability to predict neural firing patterns. Surprisingly, in terms of error consistency, CORnet-S behaves almost exactly like a standard feedforward ResNet-50 and fails to capture human-like behavior.
Key Results
- CNN vs. CNN: Consistency is very high ($\kappa \approx 0.55 \pm 0.13$).
- Human vs. Human: Consistency is moderate ($\kappa \approx 0.32 \pm 0.04$).
- CNN vs. Human: Consistency is very low. For cue-conflict and line drawings, $\kappa$ was near $0.07$, barely above chance.
- The Silhouette Exception: The gap narrowed slightly for silhouettes, but even here, humans remained far more consistent with each other than with any model.
- Accuracy vs. Consistency: Interestingly, as CNNs get more accurate on ImageNet, they do not necessarily become more human-like in their error patterns. In some cases, higher accuracy actually led to lower consistency with humans (as seen in the downward trend in the figure above).
Limitations and Future Work
The authors acknowledge that while their metric is a necessary condition for claiming "similar strategies," it is not a sufficient one (two systems could fail on the same images for different internal reasons).
Future work suggests that while recurrence (like in CORnet-S) didn't bridge the gap here, it might be more relevant for "difficult" images that require more processing time—images that weren't the primary focus of this specific trial-by-trial set. The authors envision error consistency becoming a standard benchmark for the "Brain-Score" of future AI models, forcing them to move beyond mere accuracy.
Summary updated