TL;DR
Deep neural networks trained on natural images show surprisingly similar internal representations for photos, drawings, and sketches in their early layers, mirroring human generalization abilities. However, this similarity breaks down in later layers, leading to poor classification of drawings, a problem exacerbated by a texture bias in the networks. Fine-tuning later layers on drawings largely restores performance, suggesting that core features learned from natural images are indeed generalizable.
What Problem Does This Paper Address?
Humans are remarkably adept at recognizing objects across a wide range of visual abstraction, from realistic photos to simple line drawings. Deep convolutional neural networks (CNNs), while powerful for image recognition, often struggle to generalize to such abstracted stimuli. Some studies show poor classification performance of CNNs on drawings, while others suggest similar internal representations between natural images and drawings. This paper aims to reconcile these conflicting findings by investigating how CNNs process objects at different levels of visual abstraction, comparing their internal representations and classification performance to human behavior. The core question is: to what extent do CNNs trained on natural images truly generalize to drawings, and what mechanisms (or biases) hinder or facilitate this generalization?
Methods
The researchers employed a multi-pronged approach:
- Representational Similarity Analysis (RSA): This core technique was used to compare the internal representations of CNNs and humans. RDMs (Representational Dissimilarity Matrices) were computed for each layer of the network and for human similarity judgments. These RDMs capture the pairwise similarity (or dissimilarity) between representations of different object images. Spearman rank correlations between these RDMs were used to quantify similarity across different types of depiction (photos, drawings, sketches) and between the network and human representations.
- Classification Analysis: Standard classification tasks were performed to evaluate the network's ability to categorize objects from photos, drawings, and sketches. This included intra-domain (e.g., training on photos, testing on photos) and cross-domain (e.g., training on photos, testing on drawings) classification.
- Fine-tuning: To investigate the role of later network layers and potential biases, specific layers of a pre-trained CNN were fine-tuned on a dataset of drawings.
- Multidimensional Scaling (MDS): Used for visualizing the high-dimensional RDMs in a 2D space to illustrate clustering patterns of representations.
Datasets
- Custom Stimulus Set: A dataset of 42 object categories (21 manmade, 21 natural) was created, with each object depicted in three levels of abstraction: natural photos, line drawings, and sketches. This resulted in a total of 126 stimuli.
- ImageNet: The VGG-16 model was pre-trained on the ImageNet dataset (specifically, the ILSVRC2012 dataset, which contains approximately 1.4 million images).
- ImageNet-Sketch: A dataset used for fine-tuning, consisting of approximately 50 training examples of drawings for each of the 1000 ImageNet classes (around 51,000 training images). This dataset is significantly smaller than the original ImageNet training set.
- Human Behavioral Data: Collected via online experiments (Amazon Mechanical Turk) for labeling tasks and a triplet odd-one-out similarity task.
Key Insights and Ideas
- Early Layers are General: The most striking finding is that early and intermediate layers of an ImageNet-trained CNN exhibit highly similar representational structures for photos, drawings, and sketches. This suggests that the fundamental features learned from natural images are already general enough to represent abstracted versions of the same objects.
- Late Layers Show Domain Bias: Despite similar early representations, this similarity breaks down in later layers. This "representational shift" leads to poor classification performance on drawings and sketches, indicating that the network's later processing stages become biased towards the statistics of natural images it was trained on.
- Texture Bias is a Culprit: The researchers pinpoint a "texture bias" in CNNs as a significant contributor to the representational divergence in later layers. CNNs tend to rely more on texture than shape for classification, which is problematic because texture is heavily altered in drawings and sketches.
- Drawings Aren't "Collapsed": While drawings and sketches cluster together in later layers, this isn't a complete "representational collapse." The network still retains high-level category information (e.g., manmade vs. natural) and the representations show similarity to human judgments, suggesting a "shift" rather than a complete loss of information.
- Fine-tuning Rescues Drawings: By fine-tuning only the later layers of the network on drawings, performance for drawings can be significantly restored. This reinforces the idea that early/intermediate features are general, and the issue lies in how these features are interpreted and mapped to categories in later stages.
Key Results
- Classification Performance:
- ImageNet-trained VGG-16 achieved high accuracy on photos (M=0.79) but very low accuracy on drawings (M=0.14) and sketches (M=0.02).
- Humans performed very well and similarly on photos (M=0.97) and drawings (M=0.96), with a slight drop for sketches (M=0.90).
- The network performed significantly worse than humans across all depiction types.
- Representational Similarity (RSA):
- High similarity between photos, drawings, and sketches was observed in early and intermediate layers of the ImageNet-trained VGG-16.
- This similarity dropped sharply in later layers.
- Drawing and sketch representations clustered together more than with photos in later layers.
- A shape-biased CNN (trained on stylized ImageNet) showed attenuated drops in similarity in later layers and improved drawing performance.
- Fine-tuning the later layers of the ImageNet-trained VGG-16 restored representational similarity for drawings and sketches.
- Fit to Human Behavior:
- The ImageNet-trained VGG-16 showed the best fit to human behavior for photos, with lower but significant fits for drawings and sketches.
- The shape-biased CNN and the fine-tuned network showed improved fits to human behavior for drawings and sketches.
Limitations and Future Work
- Sketch Performance Remains Poor: Even after fine-tuning, performance on highly abstracted sketch images remained poor. The authors suggest that the ImageNet-Sketch dataset might be more suitable for drawings than sketches, and that highly abstracted images might require different recognition mechanisms than those used for natural images and drawings.
- Architecture vs. Training: While the study disentangled the roles of architecture and training, it notes that the exact contribution of different layer types (e.g., fully connected layers) and alternative architectures warrants further investigation.
- Human Similarity Task Nuances: The triplet task used to measure human similarity allows participants to use information (like context) that CNNs might not capture, potentially leading to differences in how similarity is judged.
- Future Directions: The authors suggest exploring networks with different architectures and processing mechanisms, as well as investigating models that more closely match the human brain's structure and learning rules. They also highlight the potential of new models incorporating natural language supervision for achieving robust generalization.
Summary updated