TL;DR
This paper uses deep convolutional neural networks (CNNs) as computational models to understand how the human visual system processes information. By mapping the predictive power of different CNN layers to brain activity, they demonstrate a hierarchical organization in the visual cortex, mirroring the progression of feature abstraction in CNNs. This approach offers a novel way to synthesize and understand brain responses to complex visual stimuli.
What Problem Does This Paper Address?
The human visual system is incredibly adept at processing complex visual information, from recognizing simple edges to identifying intricate objects and scenes. While neuroscience has made strides in understanding specific visual areas and their functions (e.g., retinotopy, face recognition), a unified framework for how these different processing stages interact and build upon each other has been elusive. Traditional fMRI studies often rely on carefully crafted stimuli and simple contrasts, which may not fully capture the richness of naturalistic vision. This paper aims to bridge this gap by using deep CNNs, which are state-of-the-art in computer vision, as a computational model to predict and understand human brain activity across various visual processing stages. They hypothesize that the hierarchical structure of CNNs, designed for object recognition, might mirror the hierarchical organization of the human visual cortex.
Methods
The core of this research lies in using deep convolutional neural networks (CNNs) as "encoding models." Specifically, they employed the OverFeat CNN architecture, known for its performance in object recognition tasks. The key idea is to treat each layer of the CNN as a potential predictor of brain activity.
- Feature Extraction: The OverFeat network was fed with visual stimuli (images and videos). Each layer of the network extracts increasingly abstract features from the input.
- Predictive Modeling: For each layer of the CNN, a linear predictive model was trained to predict the BOLD fMRI signal in individual voxels of the human visual cortex. This means they learned a mapping from the CNN's feature representations at a given layer to the observed brain activity.
- Cross-Validation and Scoring: The predictive performance of each layer's model was evaluated using cross-validation on held-out data. A key metric used was the predictive $r^2$ score (for images) and correlation score (for videos), indicating how well the CNN layer's features could explain the brain activity in a given voxel.
- Score Fingerprints: For each voxel, a "score fingerprint" was created, representing the predictive scores from all layers. This fingerprint reveals which CNN layers best predict the activity of that specific voxel.
- Synthesis of Experiments: The trained predictive models were used as a "forward model" to synthesize brain activation maps for novel stimuli. This allowed them to test if the model could reproduce known fMRI experimental results, such as retinotopy and face/place contrasts.
Datasets
The study utilized two primary fMRI datasets:
- Still Images Dataset: This dataset, originating from [17] and [18], consists of 1750 grayscale natural images presented to subjects. This dataset was used for training and evaluating the predictive models.
- Video Dataset: This dataset, previously used in [19] and [21], comprises movie trailers and wildlife documentaries cut into short blocks. This was used to test the generalizability of the findings across different stimulus types.
Both datasets provided functionally localized regions of interest (ROIs) such as V1, V2, V3, V4, V3A, V3B, and LOC, which were crucial for analyzing layer preferences across different visual areas.
Key Insights and Ideas
- Hierarchical Representation: The study strongly supports the idea that the human visual cortex is organized hierarchically, with early visual areas (like V1) being best explained by lower-level CNN features (e.g., edge detectors), and progressively higher visual areas being better explained by more abstract, higher-level features from deeper CNN layers.
- Ventral and Dorsal Stream Progression: This hierarchical progression is not uniform; it follows distinct pathways. The ventral stream (involved in object recognition) shows a clear trend from low-level to high-level representations, while the dorsal stream (involved in spatial processing) also exhibits this trend, though potentially with different dynamics.
- CNNs as Models of Brain Function: The success in predicting brain activity across multiple visual areas and even synthesizing realistic fMRI contrasts suggests that deep CNNs, particularly those trained for object recognition, capture fundamental principles of how the human visual system represents information.
- Generalization Across Paradigms: A significant finding is that the CNN-based model, trained on natural images, could generalize to predict brain activity for stimuli from entirely different experimental paradigms (retinotopy, face/place contrasts) and even for video stimuli, demonstrating the universality of the learned representations.
- Synthesis as a Validation Tool: Using the trained models to synthesize brain activity for known fMRI experiments (like retinotopy and face/place contrasts) proved to be a powerful validation method, showing that the model could accurately replicate established neuroscientific findings.
Key Results
- Layer Preference Gradients: For each voxel, a "score fingerprint" showed a clear progression of predictive power across CNN layers, with V1 peaking at layer 1, V2/V3 peaking around layer 2, and V4/LOC showing preference for later layers.
- Spatial Mapping of Layer Preference: "Best model per voxel" maps revealed a consistent organization, with medial occipital regions (V1) best explained by early layers, and ventral/dorsal pathways showing a progression towards later layers.
- Successful Synthesis of fMRI Experiments:
- Retinotopy: The synthesized retinotopic maps accurately reproduced known visual area boundaries and angle inversions, aligning well with established retinotopic mapping results.
- Face/Place Contrast: Synthesized contrasts for faces vs. places closely matched ground-truth fMRI contrasts, highlighting known face-selective (lateral occipital) and place-selective (dorsal) regions. This synthesis was robust even with stimuli from a different dataset and was shown to be driven by high-level features, as using only the first CNN layer failed to produce meaningful contrasts.
- Robustness to Stimulus Differences: The model's ability to generalize and synthesize accurate brain activity for stimuli with different statistics and framing (e.g., from the Haxby dataset) compared to the training data underscores the model's capture of high-level invariant representations.
Limitations and Future Work
- High Dimensionality and Correlations: The authors acknowledge that the forward modeling pipeline suffers from high dimensionality and strong correlations between layers, which can make it difficult to draw definitive conclusions.
- Data Limitations: The ability to train predictive models is constrained by the availability of sufficient image/brain image pairs. For complex representations, more data might be needed.
- Spatiotemporal Operations (Video Data): The treatment of video data was limited to a frame-by-frame analysis. Future work should address the predictive capacity of spatiotemporally informed video analysis networks, as visual neurons perform dynamic operations.
- Cortico-cortical Connectivity: The current model focuses on predicting activity within visual areas. Future work could exploit this fine-grained forward model to explore cortico-cortical connectivity, predicting activity in upstream areas based on predictions from preceding areas.
- Reproducing More Contrasts: The authors suggest continuing to test known fMRI experiments to identify discrepancies between predicted and ground-truth contrasts, which could reveal limitations in the CNN's representational capacity or data availability.
- Experiment Design: The validated forward model could be used in an "explorative mode" to test new visual experimental paradigms and inform the design of future experiments. Discrepancies could guide refinement of the quantitative model.
Summary updated