A Convolutional Neural Network Interpretable Framework for Human Ventral Visual Pathway Representation

Mufan Xue, Xinyu Wu, Jinlong Li, Xuesong Li, Guoyuan YangAAAI 2024 · DOI

Introduction

Convolutional Neural Networks (CNNs) have emerged as the leading computational models for predicting neural activity in the human ventral visual pathway. However, these models are often criticized as "black boxes," providing high predictive accuracy without offering clear insights into the specific visual features that drive neural responses. To address this gap, researchers have developed the CNN Interpretable Framework (CNN-IF), a novel approach that combines high-performance encoding with biologically plausible interpretability.

Methods: The CNN-IF Architecture

The CNN-IF bridges artificial and biological vision by integrating several key components designed for transparency. At its core is the feature-weighted receptive field (fwrf) model. Unlike standard linear mappings, the fwrf separates the spatial component ("where" a voxel looks) from the feature component ("what" it sees). This separation allows for the estimation of spatial pooling fields and feature weights that align more closely with biological neural properties.

Overview of the CNN-IF architecture, including spatial pooling fields, upsampling, and the network dissection process for voxel mapping.

The framework employs a "Network Dissection" protocol to decode the internal representations of the CNN. By utilizing the Broden dataset—which contains pixel-level labels for objects, parts, and materials—the researchers quantified the alignment between specific CNN units and visual concepts using Intersection over Union (IoU) metrics. These units were then mapped to individual voxels in the brain, creating a direct correspondence between artificial neurons and fMRI-measured brain activity.

Data and Training

The framework was validated using the Natural Scenes Dataset (NSD), a high-quality 7T fMRI dataset where participants viewed thousands of images from the COCO dataset. The researchers tested two primary training regimes:

  1. Goal-driven models: Pre-trained on ImageNet for object classification.
  2. Data-driven models: Trained directly on fMRI data to predict brain responses from scratch.

Key Insights and Results

Hierarchical Alignment and ROI Selectivity

The study confirmed a robust hierarchical alignment between the CNN layers and the ventral visual pathway. Early layers (conv1-conv3) best predicted activity in the early visual cortex (V1-V3), while deeper layers (conv5-fc8) aligned with high-level category-selective regions.

Hierarchical alignment and concept selectivity across visual ROIs for different model training regimes.

As shown in Figure 4, specific regions of interest (ROIs) exhibited distinct concept selectivities. For instance, the Fusiform Face Area (FFA) and Extrastriate Body Area (EBA) both showed high selectivity for the "person" concept, though they specialized in faces and body parts, respectively. The Retrosplenial Cortex (RSC), conversely, showed a preference for "wall" and "building" concepts, aligning with its known role in scene processing.

The "Person" Concept Gradient

A significant finding was the evolution of complex concepts across the network hierarchy. While early layers might detect low-level geometric features that contribute to a "person" (such as circular head shapes), the representation only "crystallizes" into a coherent semantic concept in higher-level areas. Quantitative analysis showed a sharp increase in the number of voxels selective for "person" as the model progressed from conv4 to conv6.

Data-Driven vs. Goal-Driven Performance

The researchers found that with sufficient fMRI training data, data-driven models could approach the predictive accuracy of goal-driven models. This suggests that the fundamental encoding principles of the visual cortex can be learned directly from neural signals, even without explicit object-recognition labels.

Discussion and Limitations

While CNN-IF provides unprecedented transparency, it is constrained by the semantic labels available in the dissection datasets. For example, the model identified "building" features in the Visual Word Form Area (VWFA), likely because the Broden dataset lacked specific "text" or "word" labels, leading to a semantic mismatch. Additionally, the high data requirements for data-driven models remain a challenge for smaller-scale fMRI studies.

In conclusion, the CNN-IF represents a significant step toward interpretable neuro-AI, offering a framework that maintains high predictive power while allowing researchers to visualize and quantify the specific visual concepts represented within the human brain.

Summary updated

BibTeX
@inproceedings{xue2024convolutional,
  author = {Mufan Xue and Xinyu Wu and Jinlong Li and Xuesong Li and Guoyuan Yang},
  title = {{A Convolutional Neural Network Interpretable Framework for Human Ventral Visual Pathway Representation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28461},
}
Figure