TL;DR
This paper introduces "MindSet: Vision," a comprehensive toolbox of 30 psychological experiments designed to rigorously test deep neural networks (DNNs) against human visual perception. By systematically manipulating stimuli and employing diverse testing methods, MindSet aims to bridge the gap between computational models and human vision, revealing critical areas where current DNNs fall short.
What Problem Does This Paper Address?
The alignment between deep neural networks (DNNs) and human vision is a hot topic, with many DNNs claiming to be state-of-the-art models of human object recognition. However, current benchmarks often rely on observational data (e.g., brain responses to natural images) that aren't designed to test specific hypotheses about how humans and DNNs perceive and identify objects. This can lead to superficial similarities being mistaken for true alignment, as DNNs might achieve good performance through different internal mechanisms. The authors argue that a more rigorous approach is needed, one that uses systematically manipulated stimuli to test specific psychological findings and reveal genuine differences in how DNNs and humans process visual information.
Methods
The core of MindSet: Vision is a collection of 30 datasets, each corresponding to a psychological finding. These datasets are designed to be highly configurable, allowing researchers to regenerate them with varying parameters. To test DNNs on these datasets, the authors provide code for three primary methods:
- Out-of-Distribution Classification: Testing DNNs trained on standard datasets (like ImageNet) on novel, out-of-distribution images derived from psychological experiments. This is particularly useful for shape and object recognition tasks.
- Similarity Judgment Analysis: This method assesses the pairwise similarity of activation patterns within DNNs when presented with image pairs. It's used to see if DNNs exhibit similar representational similarities to humans across different conditions. This can be applied even to images that a DNN cannot classify.
- Decoder Method: A small, often single-layer, "decoder" network is attached to a frozen DNN layer. This decoder is trained on a specific task (e.g., predicting object size) using non-illusory stimuli and then tested on illusory stimuli to see if the DNN exhibits human-like perceptual biases.
Datasets
MindSet: Vision includes datasets for 30 psychological findings, categorized into:
- Low and Mid-Level Vision: Covering phenomena like Weber's Law, Crowding/Uncrowding, Emergent Features, Decomposition, Relational vs. Coordinate Changes, 3D Shape Encoding, and Amodal Completion.
- Visual Illusions: Including classic illusions such as Müller-Lyer, Ponzo, Ebbinghaus, Jastrow, Tilt, Lightness Contrast, Adelson Checker Shadow, and Thatcher Illusions.
- Shape and Object Recognition: Datasets for line drawings, dotted line drawings, silhouettes, texturized objects, embedded shapes, and the Same/Different task.
These datasets are largely procedurally generated, allowing for extensive parameter customization. They are made available under an MIT license.
Key Insights and Ideas
- The Need for "Severe" Testing: The paper strongly advocates for moving beyond observational benchmarks to "severe" tests that systematically probe specific hypotheses about visual processing. This is crucial for avoiding superficial alignment claims.
- DNNs Struggle with Core Perceptual Phenomena: Even state-of-the-art DNNs, which perform well on benchmarks like Brain-Score, often fail to replicate human-like responses on these psychologically-grounded tasks.
- Amodal Completion is Not Universally Present: While some networks show a slight tendency towards amodal completion, the effect is often minimal and inconsistent across layers, suggesting it's not a robust feature.
- Relational vs. Coordinate Changes: DNNs don't consistently show the human sensitivity to relational changes between object parts over coordinate changes, indicating a potential gap in their understanding of object structure.
- 3D Information Encoding is Limited: DNNs struggle to encode 3D shape information from 2D images as effectively as humans do, often performing worse on stimuli that rely on depth cues.
- Illusions Reveal Specific Weaknesses: DNNs show mixed results on visual illusions. Some illusions, like the Müller-Lyer, are consistently missed, while others, like the Ebbinghaus, show some effect but with often unhuman-like magnitudes or directions.
- Shape Recognition is Still a Challenge: Despite advances, DNNs still lag behind human performance in shape recognition tasks, especially with modified stimuli like texturized images or line drawings.
Key Results
The authors tested 15 DNNs on 9 of the MindSet experiments. The key takeaway is that most models struggled across many of the experiments.
- Amodal Completion: Similarity scores for occlusion vs. notched conditions were only slightly higher than baseline, with minimal and inconsistent effects across layers.
- Relational vs. Coordinate Change: Networks showed similar similarity scores for both conditions, failing to replicate human sensitivity to relational changes.
- Metric vs. Non-Accidental Properties: DNNs showed very similar similarity scores for both MP and NAP changes, unlike humans who are more sensitive to NAP changes.
- Depth Drawings: Networks often showed the opposite pattern to humans, performing better on stimuli that were less discriminable for humans based on 3D cues.
- Illusions (Ebbinghaus, Müller-Lyer, Jastrow): Decoders often failed to generalize from training to test stimuli. When effects were observed, magnitudes were often unhuman-like, and directions were sometimes reversed (e.g., Ebbinghaus).
- Shape Recognition (Line drawings, Dotted, Silhouettes, Texturized): DNNs performed significantly worse than humans (who achieve >90% accuracy) on all these tasks, with performance dropping further on texturized stimuli.
Limitations and Future Work
- Limited Scope of Testing: Only 9 out of 30 experiments were tested on DNNs. The authors acknowledge that DNNs can likely be built to succeed on these tasks, and the toolkit is intended to motivate further research.
- Lack of Behavioral Datasets: While the stimuli are designed to test specific hypotheses, comprehensive behavioral datasets for all 30 experiments are not yet provided. However, the authors argue that the existing stimuli are sufficient for testing.
- Superficial Analysis: The analysis of models on the 9 experiments was superficial. More detailed analyses are needed to uncover further strengths and weaknesses.
- Future Work: The authors envision the toolkit facilitating more in-depth research into DNN-brain alignment, encouraging exploration of other psychological domains (memory, language), and driving the development of more human-like DNN models. They also suggest that researchers can use the toolkit to explore different data augmentation strategies and decoder architectures.
Summary updated