TL;DR
This study investigates how humans and deep neural networks represent complex visual scenes. They found that while human behavior relies on both functional (action-related) and visual features, the brain's scene-selective areas primarily represent visual features captured by deep neural networks, showing a dissociation between behavioral and neural representations.
What Problem Does This Paper Address?
Understanding how our brains process complex visual scenes is challenging due to the inherent correlations between different scene properties (e.g., objects, layout, semantics). For instance, a forest might be characterized by specific textures, spatial arrangements, and the presence of trees, making it hard to isolate the contribution of each factor to neural representations. This paper aims to disentangle these contributions by comparing how functional (action-based), object-based, and deep neural network (DNN) visual features explain human behavior and brain activity when viewing scenes.
Methods
The researchers employed a Representational Similarity Analysis (RSA) approach. They first curated a set of 30 scene categories, carefully selecting them to minimize correlations between three feature models:
- Functional Model: Based on human-assigned labels of actions that can be performed in a scene.
- Object Model: Based on human-assigned labels of objects present in a scene.
- DNN Features: Derived from a high-level layer (fc7) of a pre-trained AlexNet convolutional neural network.
They then collected fMRI data from participants viewing these scenes while performing an unrelated task. Behavioral data was collected using a multi-arrangement task where participants organized scenes based on perceived similarity. RSA was used to compare the similarity structures (Representational Dissimilarity Matrices - RDMs) derived from the three feature models with the behavioral and fMRI RDMs. Variance partitioning was used to quantify the unique and shared contributions of each model.
Datasets
- SUN database: Used as the source for scene categories.
- ImageNet: The dataset used to pre-train the AlexNet DNN model.
- Custom Stimulus Set: A curated set of 30 scene categories, selected to minimize inter-model correlations, used for fMRI and behavioral experiments.
Key Insights and Ideas
- Behavioral vs. Neural Dissociation: The most striking finding is the divergence between what drives human behavior and what is represented in scene-selective brain regions. Humans use both functional and visual information for scene categorization, but the brain seems to prioritize visual features.
- DNNs as Powerful Scene Encoders: High-level DNN features proved to be remarkably good at explaining neural activity in scene-selective areas (PPA and OPA), outperforming semantic (object and function) models.
- Functional Information's Role: While functional information is crucial for human behavioral scene categorization, its unique contribution is not strongly represented in the primary scene-selective cortical areas. Instead, it appears to be represented in regions outside of these core areas.
- Stimulus Selection Matters: The careful selection of stimuli to minimize feature correlations was critical for isolating the unique contributions of each model.
Key Results
- Behavioral Similarity: The functional model contributed uniquely to behavioral scene similarity (37.6% of explained variance), followed by DNN features (29.0%), and a negligible contribution from the object model (1.4%).
- fMRI Similarity (Scene-Selective Cortex): In PPA and OPA, DNN features uniquely explained the majority of the variance (71.1% and 86.9%, respectively). The functional model contributed very little unique variance (<1% in PPA, 2.6% in OPA). The object model also had minimal unique contribution.
- Functional Information Outside Scene-Selective Cortex: Whole-brain searchlight analyses revealed clusters correlating with the functional model in bilateral ventral and left lateral occipito-temporal cortex, potentially overlapping with face, body, and motion-selective regions. However, these regions did not correlate with the behavioral scene categorization task.
- Task Influence: An experiment with a covert naming task (Experiment 2) showed increased correlations with DNN features in PPA and OPA, but did not significantly boost the contribution of the functional model to scene-selective cortex.
Limitations and Future Work
- Limited DNN Models: The study acknowledges that they did not exhaustively test all possible DNN models. Future research could explore other architectures (e.g., VGG, GoogleNet, ResNet) or DNNs trained on different datasets (e.g., Atomic Visual Actions, Moments in Time) that might better capture functional information.
- Task Specificity: The dissociation between behavioral and neural representations might be influenced by task demands. The authors suggest that explicit contrastive tasks or tasks requiring consideration of functional aspects might reveal different neural representations.
- Distributed Representations: The findings suggest that scene understanding is not solely confined to scene-selective cortex. Future work should investigate a broader network of brain regions and explore how different tasks and feature models interact to support complex visual cognition.
- Data Availability: The authors have made their fMRI and behavioral data publicly available to facilitate future research.
Summary updated