Abstract
Human subjects are extremely efficient at categorizing natural scenes, despite the fact that different classes of natural scenes often share similar image statistics. Thus far, however, it is unknown where and how complex natural scene categories are encoded and discriminated in the brain. We used functional magnetic resonance imaging (fMRI) and distributed pattern analysis to ask what regions of the brain can differentiate natural scene categories (such as forests vs mountains vs beaches). Using completely different exemplars of six natural scene categories for training and testing ensured that the classification algorithm was learning patterns associated with the category in general and not specific exemplars. We found that area V1, the parahippocampal place area (PPA), retrosplenial cortex (RSC), and lateral occipital complex (LOC) all contain information that distinguishes among natural scene categories. More importantly, correlations with human behavioral experiments suggest that the information present in the PPA, RSC, and LOC is likely to contribute to natural scene categorization by humans. Specifically, error patterns of predictions based on fMRI signals in these areas were significantly correlated with the behavioral errors of the subjects. Furthermore, both behavioral categorization performance and predictions from PPA exhibited a significant decrease in accuracy when scenes were presented up-down inverted. Together these results suggest that a network of regions, including the PPA, RSC, and LOC, contribute to the human ability to categorize natural scenes.
Introduction
Humans possess a remarkable ability to categorize visual scenes—such as identifying a "beach" or a "mountain"—in under 100 milliseconds. This speed is maintained even when images are novel or cluttered. Despite this efficiency, the underlying neural mechanisms have remained elusive. Traditional fMRI analyses, which average activity across large brain regions, often miss the subtle, distributed "fingerprints" that define complex categories. Moreover, the presence of information in a brain region does not necessarily mean the brain utilizes that information for decision-making.
This study addresses these gaps by investigating where category information is stored and whether this information correlates with human behavior. The researchers employed a multi-faceted approach combining fMRI with behavioral testing and machine learning. Subjects viewed 720 color images across six categories: beaches, buildings, forests, highways, industry, and mountains. Using Multivoxel Pattern Analysis (MVPA), a Support Vector Machine (SVM) classifier was trained to identify categories based on distributed patterns of activity. Outside the scanner, subjects performed rapid categorization tasks to generate "confusion matrices," identifying which categories were most frequently mistaken for one another.
Results
The SVM successfully decoded categories from several regions: PPA (31%), RSC (27%), V1 (26%), and LOC (24%), all significantly above the 16.7% chance level. However, the presence of information did not always translate to behavioral relevance.
The Behavioral Mirror
A key finding was that the "mistakes" made by the machine learning decoder trained on brain activity mirrored the behavioral errors made by humans in higher-level visual areas.
As seen in the confusion matrices, the PPA's error patterns (e.g., confusing buildings with industrial sites) closely align with human behavioral errors. In contrast, V1's errors correlated more with low-level physical image similarity rather than human perception.
Quantitative Correlations Across Regions
The correlation between decoder errors and human errors was strongest in the PPA ($r = 0.57$) and LOC ($r = 0.42$).
These scatter plots demonstrate that while multiple regions contain category information, the error structures in the PPA, RSC, and LOC are most consistent with human behavior. The FFA showed no significant decoding, reinforcing its specialization for faces. Additionally, a whole-brain searchlight analysis identified category information in Broca’s area, potentially reflecting subjects' internal naming of categories.
The Role of Spatial Layout
The study also explored the "inversion effect." Flipping images upside down disrupts global spatial layout. This manipulation led to a significant drop in both human categorization accuracy and PPA decoding accuracy, suggesting the PPA relies heavily on global layout for scene identification.
Discussion
These results suggest that scene categorization is handled by a distributed hierarchy. V1 processes low-level features, the LOC likely identifies constituent objects, and the PPA/RSC network integrates global spatial context.
While the correlations are compelling, the authors acknowledge that fMRI is inherently correlational and cannot definitively prove causality. Furthermore, the temporal gap between fMRI signals (seconds) and rapid categorization (milliseconds) remains a challenge. Future research using high-temporal resolution techniques like EEG or MEG could further elucidate the directional flow of information between these regions—specifically whether the PPA provides "top-down" gist information to help the LOC identify objects, or vice versa.
Summary updated