TL;DR
This paper introduces "brain dissection," a novel method that combines fMRI data with explainable AI techniques to map how different brain regions process spatial features in natural images. By training neural networks to predict brain responses and then dissecting these networks, the researchers reveal distinct spatial selectivity patterns across the visual cortex, offering a more granular understanding of visual processing.
What Problem Does This Paper Address?
Understanding how the human brain processes visual information, especially in higher visual areas, is a significant challenge. While deep neural networks (DNNs) have shown promise in explaining cortical responses, their "black box" nature makes them difficult to interpret. Traditional methods for studying brain function often rely on carefully crafted stimuli or focus on broad categories, limiting our ability to understand how the brain processes the rich, complex information present in natural scenes. Specifically, it's unclear how different visual streams and sub-regions within them specialize in processing intermediate spatial features like depth, surface orientation, and curvature, which are crucial for navigating and interacting with the world.
Methods
The core innovation is "brain dissection," an adaptation of the "network dissection" technique from explainable AI. Here's how it works:
- fMRI Response-Optimized Network Training: For specific regions of interest (ROIs) in the visual cortex, researchers train a convolutional neural network (CNN) to predict the fMRI response of individual voxels to natural images. Crucially, all voxels within an ROI share a common CNN backbone, but each voxel has its own learnable "spatial weights" (indicating where in the image it's most responsive) and "feature weights" (indicating what visual information it's most sensitive to). This training is done using the Natural Scenes Dataset (NSD).
- Brain Dissection: After training, the spatial weights are discarded, and only the feature weights are used. For each voxel, these feature weights are applied to the CNN's output activations for a given image, creating a "voxel-specific feature map." This map highlights the image regions that most strongly activate that voxel. By thresholding this map, researchers obtain a binary mask indicating the most relevant parts of the image for that voxel.
- Measure Analysis: The generated masks are then compared against ground-truth spatial measures (like depth maps, surface normals, curvature, and shading) or object/attribute annotations from datasets like Places365 and GQA. This comparison allows researchers to quantify the selectivity of each voxel for specific spatial properties or concepts.
This approach is "hypothesis-neutral," meaning it doesn't start with pre-conceived notions about what each brain region should process.
Datasets
- Natural Scenes Dataset (NSD) [5]: Used for training the fMRI response-optimized neural networks. This dataset contains high-resolution 7T fMRI responses to a large collection of naturalistic images from Microsoft COCO, collected from 8 subjects.
- Places365 Dataset [41]: Used for evaluating the selectivity of brain regions for spatial measures like depth, surface normals, curvature, and shading.
- GQA Dataset [46]: Used for evaluating selectivity related to object relationships, attributes, and categories. This dataset is based on the Visual Genome dataset [47] and provides scene graph annotations.
Key Insights and Ideas
- Beyond Category Selectivity: The brain doesn't just categorize objects; it processes a rich set of intermediate spatial features that inform our interaction with the environment.
- Functional Segregation in Visual Streams: Different visual streams (ventral, lateral, parietal) and sub-regions within them (e.g., OPA, PPA, RSC) show distinct preferences for spatial properties, suggesting specialized roles.
- Spatial Coordinate Systems: Differences in surface normal preferences hint at the use of different spatial coordinate systems (e.g., egocentric vs. allocentric) in various brain regions.
- Scene Network Specialization: Even within the "scene network" (RSC, OPA, PPA), there are clear distinctions: RSC is more attuned to distant, outdoor scenes, while OPA and PPA focus more on local, indoor elements and verticality.
- Topographical Organization: Spatial preferences are not random but show a degree of topographical organization within ROIs, meaning neighboring voxels tend to have similar preferences.
Key Results
- Depth Preferences: Ventromedial and Medial areas show strong preferences for far depths, while Ventrolateral and Lateral areas prefer near depths. A gradient exists from near to far depths across Ventral areas.
- Surface Normals: Lateral and Ventrolateral regions prefer surface normals pointing back at the camera (egocentric), while Parietal and Medial areas show preferences for "up" and "left/right" normals. RSC prefers "right/left" normals, suggesting vertical structures relevant to navigation. OPA and PPA favor "up" normals.
- Curvature and Shading: Ventrolateral and Lateral areas prefer curvier objects and lighter shading, potentially related to object recognition and interaction. Parietal and Ventromedial areas prefer flatter surfaces and darker shading, possibly for scene layout processing.
- Scene Regions: RSC shows a strong preference for far depths and outdoor scenes, aligning with its role in navigation. OPA shows selectivity for indoor elements, while PPA has mixed indoor/outdoor preferences.
- Object Relations/Attributes: Parietal areas favor inanimate objects and spatial relations (e.g., "in front of"), Lateral areas prefer animate objects and person-centric relations (e.g., "wearing"), and Ventral areas show a mix.
Limitations and Future Work
- Voxel Isolation: Network dissection interprets each voxel independently, potentially missing selectivity that emerges from the interaction of groups of voxels. This aligns with voxel-wise modeling but not multivariate pattern analysis.
- Correlations in Natural Images: Natural images have inherent correlations between categories and their spatial attributes. While the study focuses on ROIs known to encode spatial properties, this correlation can still influence findings.
- Future Work: The authors suggest that brain dissection offers a powerful, data-driven approach to uncovering insights into the entire visual processing hierarchy, from early visual processing to high-level semantic representations. This method can be extended to explore other intermediate features and brain regions.
Summary updated