Fourier power, subjective distance, and object categories all provide plausible models of BOLD responses in scene-selective visual areas

Mark D. Lescroart, Dustin E. Stansbury, Jack L. GallantFrontiers Comput. Neurosci 2015 · OpenReview

Abstract

Perception of natural visual scenes activates several functional areas in the human brain, including the Parahippocampal Place Area (PPA), Retrosplenial Complex (RSC), and the Occipital Place Area (OPA). It is currently unclear what specific scene-related features are represented in these areas. Previous studies have suggested that PPA, RSC, and/or OPA might represent at least three qualitatively different classes of features: (1) 2D features related to Fourier power; (2) 3D spatial features such as the distance to objects in a scene; or (3) abstract features such as the categories of objects in a scene. To determine which of these hypotheses best describes the visual representation in scene-selective areas, we applied voxel-wise modeling (VM) to BOLD fMRI responses elicited by a set of 1386 images of natural scenes. VM provides an efficient method for testing competing hypotheses by comparing predictions of brain activity based on encoding models that instantiate each hypothesis. Here we evaluated three different encoding models that instantiate each of the three hypotheses listed above. We used linear regression to fit each encoding model to the fMRI data recorded from each voxel, and we evaluated each fit model by estimating the amount of variance it predicted in a withheld portion of the data set. We found that voxel-wise models based on Fourier power or the subjective distance to objects in each scene predicted much of the variance predicted by a model based on object categories. Furthermore, the response variance explained by these three models is largely shared, and the individual models explain little unique variance in responses. Based on an evaluation of previous studies and the data we present here, we conclude that there is currently no good basis to favor any one of the three alternative hypotheses about visual representation in scene-selective areas. We offer suggestions for further studies that may help resolve this issue.

Introduction

When we navigate a landscape or enter a room, specific regions in the human brain—the Parahippocampal Place Area (PPA), Retrosplenial Complex (RSC), and Occipital Place Area (OPA)—show robust activation. For decades, a central debate in visual neuroscience has centered on what exactly these "scene-selective" areas are representing.

Theories generally fall into three camps:

  1. 2D Features: Responses are driven by basic image properties like textures, edges, and spatial frequencies (Fourier power).
  2. 3D Spatial Features: Representation focuses on the geometry of a scene, such as the subjective distance to objects or the "openness" of a space.
  3. Abstract Categories: These areas represent high-level semantics—identifying the scene as a "kitchen," "forest," or "building."

Historically, studies have used highly controlled stimuli to test these hypotheses in isolation. This study by Lescroart, Stansbury, and Gallant utilizes a massive set of natural images and voxel-wise modeling to see if these competing theories can be disentangled in a real-world context.

Methods: Voxel-wise Modeling and Feature Spaces

The researchers employed Voxel-wise Modeling (VM), a framework that maps complex stimulus features to brain activity at the level of individual functional units (voxels). They constructed three distinct encoding models to represent the competing hypotheses:

  • Fourier Power Model: Captured 2D spatial frequency and orientation (9 channels).
  • Subjective Distance Model: Based on human ratings of the distance to the "main content" of each image (5 channels).
  • Object Category Model: Based on human-assigned labels for 19 categories (e.g., "edifice," "land," "human").
Voxel-wise modeling workflow illustrating the three feature spaces and the prediction pipeline.

Models were fit using linear regression on a training set of 1,260 natural photographs and validated on a withheld set of 126 images (repeated 12 times to ensure a high signal-to-noise ratio). To ensure the reliability of the results, the researchers evaluated response variability and noise ceilings across voxels to select those with sufficient signal for further analysis.

Response variability in voxels with different noise ceilings, showing mean responses to validation images.

Results: The Correlation Trap and Shared Variance

All three models significantly predicted BOLD responses across all scene-selective areas. While the Object Category model generally provided the highest raw prediction scores ($r$), especially in the PPA and OPA, the study's critical insight came from variance partitioning.

In the natural world, 2D, 3D, and semantic features are rarely independent. For example, a "far away" scene (3D) often contains a horizon line (2D Fourier property) and the category "sky" (Semantic). The researchers found that in the PPA, RSC, and OPA, the vast majority of the variance explained by the models was shared. If the 2D and 3D information is removed, the Object Category model loses most of its predictive power, and vice versa. This suggests that while all three models are "plausible," none can be definitively crowned as the "true" underlying representation based on current natural image datasets.

Conclusion and Future Directions

The study warns against the common practice of claiming a brain area "represents X" simply because a model of X successfully predicts activity. Because natural features are so highly correlated, multiple models can appear successful simultaneously. To resolve this, future research must move toward "decorrelated" stimulus sets—synthesizing or searching for images where, for instance, the 3D layout is complex but the 2D Fourier power is low, or where the semantic category is mismatched with the spatial geometry. Disentangling these features is the next frontier for scene neuroscience.

Summary updated

Abstract

p>Perception of natural visual scenes activates several functional areas in the human brain, including the Parahippocampal Place Area (PPA), Retrosplenial Complex (RSC), and the Occipital Place Area (OPA). It is currently unclear what specific scene-related features are represented in these areas. Previous studies have suggested that PPA, RSC, and/or OPA might represent at least three qualitatively different classes of features: (1) 2D features related to Fourier power; (2) 3D spatial features such as the distance to objects in a scene; or (3) abstract features such as the categories of objects in a scene. To determine which of these hypotheses best describes the visual representation in scene-selective areas, we applied voxel-wise modeling (VM) to BOLD fMRI responses elicited by a set of 1386 images of natural scenes. VM provides an efficient method for testing competing hypotheses by comparing predictions of brain activity based on encoding models that instantiate each hypothesis. Here we evaluated three different encoding models that instantiate each of the three hypotheses listed above. We used linear regression to fit each encoding model to the fMRI data recorded from each voxel, and we evaluated each fit model by estimating the amount of variance it predicted in a withheld portion of the data set. We found that voxel-wise models based on Fourier power or the subjective distance to objects in each scene predicted much of the variance predicted by a model based on object categories. Furthermore, the response variance explained by these three models is largely shared, and the individual models explain little unique variance in responses. Based on an evaluation of previous studies and the data we present here, we conclude that there is currently no good basis to favor any one of the three alternative hypotheses about visual representation in scene-selective areas. We offer suggestions for further studies that may help resolve this issue.</p>
BibTeX
@inproceedings{lescroart2015fourier,
  author = {Mark D. Lescroart and Dustin E. Stansbury and Jack L. Gallant},
  title = {{Fourier power, subjective distance, and object categories all provide plausible models of BOLD responses in scene-selective visual areas}},
  booktitle = {Frontiers Comput. Neurosci},
  year = {2015},
  url = {https://openreview.net/forum?id=wMZr0CulaY},
}
Figure