TL;DR
This paper investigates whether deep neural networks (DNNs) can generalize object recognition as well as humans. They found that while DNNs match or exceed human performance on image perturbations (like blurring or shuffling), humans still outperform DNNs on style variations (like paintings or cartoons). However, by training a simple linear decoder on DNN features from all styles, DNNs could match or surpass human performance, suggesting DNNs' internal representations are rich enough for broad generalization.
What Problem Does This Paper Address?
A long-standing question in AI and neuroscience is whether deep neural networks, despite their impressive performance on benchmark tasks like ImageNet, truly generalize like humans do. Humans are remarkably adept at recognizing objects even when presented with novel variations – think of recognizing a car in a sketch, a painting, or a blurry photograph. DNNs, on the other hand, are often perceived as being "domain-specific," meaning their performance can degrade significantly when faced with inputs that differ from their training data. This paper aims to rigorously compare human and DNN generalization abilities across two key dimensions: image perturbations (e.g., scrambling parts of an image) and changes in visual style (e.g., cartoon vs. photograph).
Methods
The researchers employed a comparative approach, testing both humans and ImageNet-trained DNNs on a core object recognition task. For DNNs, they used pre-trained ResNet-152 and PNASNet models. To assess generalization, they evaluated performance on various image transformations and styles. Crucially, they explored two training scenarios for the DNNs: training only on a subset of the data (e.g., original images) and training on all available data. They also investigated the potential of a simple linear decoder to extract object information from the DNNs' feature representations, particularly when trained on a diverse set of styles.
Datasets
The study utilized a combination of existing and custom-generated image sets:
- HvM dataset: Synthetic naturalistic images generated by placing 3D object models onto random backgrounds.
- Microsoft COCO dataset: Natural photographs of objects.
- Artistic images: Paintings, sketches, cartoons, and line drawings.
These datasets were further manipulated to create perturbed versions (e.g., block-shuffled, blurred, swirled, silhouette) and style-transferred versions. The core object categories tested were bear, elephant, person, car, dog, apple, plane, chair, zebra, and bird.
Key Insights and Ideas
- DNNs are surprisingly robust to perturbations: It was unexpected that DNNs, even without explicit training on these variations, could match or even surpass human performance on tasks involving image scrambling, blurring, and other geometric distortions.
- Humans retain an edge in style generalization: Despite DNNs' robustness to perturbations, humans still demonstrated superior generalization when faced with significant changes in visual style, such as artistic renderings. This suggests that the "essence" of an object's visual features might be more readily understood by humans across diverse artistic interpretations.
- Richness of DNN feature spaces: The finding that a simple linear decoder, when trained on features from all tested styles, could achieve human-level or better performance across all categories is a significant insight. It implies that the internal representations learned by ImageNet-trained DNNs are already highly informative and disentangled, containing the necessary information for generalization, even if the direct classifier struggles.
Key Results
- Image Perturbations: Models matched or exceeded human performance on most image perturbations, including block-shuffled, blurred, and swirled images. For example, in Figure 2a, the "train on all" plots show that models often perform better than humans on these perturbed images.
- Image Styles: Humans generally performed better than models when generalizing to unfamiliar visual styles like paintings and cartoons (Figure 2b). However, models showed comparable performance to humans on paintings, possibly due to their statistical similarity to natural photographs.
- Linear Decodability: When a linear decoder was trained on features from all image sets (including artistic styles), models could distinguish between all 10 object categories at or slightly above human level for all image styles. This "train on all" scenario in Figure 2 demonstrates the power of the underlying DNN representations.
Limitations and Future Work
The authors acknowledge that their task, while challenging, was relatively simple, involving only two choices per trial and ten object categories. They suggest that more complex tasks, such as multiple-choice or free-labeling, and more stringent evaluation metrics (like behavioral consistency) might reveal more subtle discrepancies between human and model generalization abilities. Future work could explore these more challenging scenarios and investigate how to further improve DNN generalization, potentially by leveraging the rich feature spaces identified in this study.
Summary updated