Introduction
Convolutional Neural Networks (CNNs) have emerged as the leading computational models for predicting neural activity in the human ventral visual pathway. However, these models are often criticized as "black boxes," providing high predictive accuracy without offering clear insights into the specific visual features that drive neural responses. To address this gap, researchers have developed the CNN Interpretable Framework (CNN-IF), a novel approach that combines high-performance encoding with biologically plausible interpretability.
Methods: The CNN-IF Architecture
The CNN-IF bridges artificial and biological vision by integrating several key components designed for transparency. At its core is the feature-weighted receptive field (fwrf) model. Unlike standard linear mappings, the fwrf separates the spatial component ("where" a voxel looks) from the feature component ("what" it sees). This separation allows for the estimation of spatial pooling fields and feature weights that align more closely with biological neural properties.
The framework employs a "Network Dissection" protocol to decode the internal representations of the CNN. By utilizing the Broden dataset—which contains pixel-level labels for objects, parts, and materials—the researchers quantified the alignment between specific CNN units and visual concepts using Intersection over Union (IoU) metrics. These units were then mapped to individual voxels in the brain, creating a direct correspondence between artificial neurons and fMRI-measured brain activity.
Data and Training
The framework was validated using the Natural Scenes Dataset (NSD), a high-quality 7T fMRI dataset where participants viewed thousands of images from the COCO dataset. The researchers tested two primary training regimes:
- Goal-driven models: Pre-trained on ImageNet for object classification.
- Data-driven models: Trained directly on fMRI data to predict brain responses from scratch.
Key Insights and Results
Hierarchical Alignment and ROI Selectivity
The study confirmed a robust hierarchical alignment between the CNN layers and the ventral visual pathway. Early layers (conv1-conv3) best predicted activity in the early visual cortex (V1-V3), while deeper layers (conv5-fc8) aligned with high-level category-selective regions.
As shown in Figure 4, specific regions of interest (ROIs) exhibited distinct concept selectivities. For instance, the Fusiform Face Area (FFA) and Extrastriate Body Area (EBA) both showed high selectivity for the "person" concept, though they specialized in faces and body parts, respectively. The Retrosplenial Cortex (RSC), conversely, showed a preference for "wall" and "building" concepts, aligning with its known role in scene processing.
The "Person" Concept Gradient
A significant finding was the evolution of complex concepts across the network hierarchy. While early layers might detect low-level geometric features that contribute to a "person" (such as circular head shapes), the representation only "crystallizes" into a coherent semantic concept in higher-level areas. Quantitative analysis showed a sharp increase in the number of voxels selective for "person" as the model progressed from conv4 to conv6.
Data-Driven vs. Goal-Driven Performance
The researchers found that with sufficient fMRI training data, data-driven models could approach the predictive accuracy of goal-driven models. This suggests that the fundamental encoding principles of the visual cortex can be learned directly from neural signals, even without explicit object-recognition labels.
Discussion and Limitations
While CNN-IF provides unprecedented transparency, it is constrained by the semantic labels available in the dissection datasets. For example, the model identified "building" features in the Visual Word Form Area (VWFA), likely because the Broden dataset lacked specific "text" or "word" labels, leading to a semantic mismatch. Additionally, the high data requirements for data-driven models remain a challenge for smaller-scale fMRI studies.
In conclusion, the CNN-IF represents a significant step toward interpretable neuro-AI, offering a framework that maintains high predictive power while allowing researchers to visualize and quantify the specific visual concepts represented within the human brain.
Summary updated