Deep neural networks and visuo-semantic models explain complementary components of human ventral-stream representational dynamics

Kamila Jozwik, Tim C Kietzmann, Radoslaw Martin Cichy, Nikolaus Kriegeskorte, Marieke MurOpenReview 2023 · OpenReview

Abstract

Deep neural networks (DNNs) are promising models of the cortical computations supporting human object recognition. However, despite their ability to explain a significant portion of variance in neural data, the agreement between models and brain representational dynamics is far from perfect. We address this issue by asking which representational features are currently unaccounted for in neural time series data, estimated for multiple areas of the ventral stream via source-reconstructed magnetoencephalography data acquired in human participants during object viewing. We focus on the ability of visuo-semantic models, consisting of human-generated labels of object features and categories, to explain variance beyond the explanatory power of DNNs alone. We report a gradual reversal in the relative importance of DNN versus visuo-semantic features as ventral-stream object representations unfold over space and time. Although lower-level visual areas are better explained by DNN features starting early in time (at 66 ms after stimulus onset), higher-level cortical dynamics are best accounted for by visuo-semantic features starting later in time (at 146 ms after stimulus onset). Among the visuo-semantic features, object parts and basic categories drive the advantage over DNNs. These results show that a significant component of the variance unexplained by DNNs in higher-level cortical dynamics is structured and can be explained by readily nameable aspects of the objects. We conclude that current DNNs fail to fully capture dynamic representations in higher-level human visual cortex and suggest a path toward more accurate models of ventral-stream computations.

The Gap in Computational Models of Vision

In the quest to understand human vision, researchers have increasingly turned to Deep Neural Networks (DNNs) as "silicon models" of the ventral stream—the "what" pathway of the brain. While DNNs can explain a significant portion of brain activity, the match is far from perfect. The core question this research tackles is: What exactly is missing from our current AI models of vision? Is the variance that DNNs fail to explain just "noise," or is it structured information that the human brain prioritizes? To find out, the study compared DNNs against "visuo-semantic" models—collections of human-generated labels for object parts (e.g., "eye," "wheel") and categories (e.g., "bird," "tool").

Experimental Framework and Methods

The researchers used source-reconstructed magnetoencephalography (MEG) to record brain activity from 15 participants with millisecond precision. This allowed them to track how object representations unfold across both space (different brain regions) and time.

The experimental framework: (a) stimulus set, (b) computational models, (c) RSA computation, (d) human MEG data, and (e) model fitting.

They compared two types of models:

  1. DNN Models: CORnet-Z (a simple feedforward model) and CORnet-R (which includes local recurrent connections to better mimic brain anatomy). Both were trained on ImageNet.
  2. Visuo-Semantic Models: These consisted of 229 labels generated by human observers describing the features and categories of the 92 images used in the MEG experiment.

Using Representational Similarity Analysis (RSA) and a second-level General Linear Model (GLM), the team partitioned the brain's neural activity. This allowed them to see how much unique variance was explained by the DNNs versus the human-labeled semantic features at every millisecond after an image appeared.

Key Results: A Temporal Handoff

The most striking finding is a temporal shift in model performance. Early on (~66ms), DNN features are the best predictors of activity in lower-level visual areas (V1-V3). However, as processing continues (~146ms), human-labeled semantic features take the lead in higher-level areas like the Inferior Temporal (IT) cortex.

Variance explained over time for individual participants, showing the complementary explanatory power of DNN and visuo-semantic models.
  • Lower-Level Cortex (V1-V3): DNNs significantly outperformed visuo-semantic models in the early (66–128 ms) and late (422–844 ms) phases of the response.
  • Higher-Level Cortex (IT/PHC): Visuo-semantic models took over, significantly outperforming DNNs between 146 ms and 488 ms post-stimulus.
  • Feature Drivers: Among the semantic features, object parts and basic categories were the primary drivers of the unique variance explained in high-level cortical dynamics.
  • Recurrence and the "Visual Diet": Even though the recurrent model (CORnet-R) is more biologically plausible than feedforward models, it still could not account for the unique semantic variance. This implies the gap isn't just about architecture, but what information is being extracted. The authors suggest humans learn from a rich, multi-modal world where parts and categories have functional and evolutionary significance, unlike the static ImageNet labels used for DNN training.

Future Directions

The primary limitation is that visuo-semantic models are not currently image-computable—they require human labels. The ultimate goal is to create a DNN that can compute these semantic features directly from pixels. To achieve this, the authors suggest enriching the "visual diet" of AI models with more ecologically valid datasets (like video) and incorporating human-like learning objectives that encourage the learning of object parts and broader semantic relationships.

Summary updated

Abstract

Deep neural networks (DNNs) are promising models of the cortical computations supporting human object recognition. However, despite their ability to explain a significant portion of variance in neural data, the agreement between models and brain representational dynamics is far from perfect. We address this issue by asking which representational features are currently unaccounted for in neural time series data, estimated for multiple areas of the ventral stream via source-reconstructed magnetoencephalography data acquired in human participants (nine females, six males) during object viewing. We focus on the ability of visuo-semantic models, consisting of human-generated labels of object features and categories, to explain variance beyond the explanatory power of DNNs alone. We report a gradual reversal in the relative importance of DNN versus visuo-semantic features as ventral-stream object representations unfold over space and time. Although lower-level visual areas are better explained by DNN features starting early in time (at 66 ms after stimulus onset), higher-level cortical dynamics are best accounted for by visuo-semantic features starting later in time (at 146 ms after stimulus onset). Among the visuo-semantic features, object parts and basic categories drive the advantage over DNNs. These results show that a significant component of the variance unexplained by DNNs in higher-level cortical dynamics is structured and can be explained by readily nameable aspects of the objects. We conclude that current DNNs fail to fully capture dynamic representations in higher-level human visual cortex and suggest a path toward more accurate models of ventral-stream computations.
BibTeX
@misc{jozwik2023deep,
  author = {Kamila Jozwik and Tim C Kietzmann and Radoslaw Martin Cichy and Nikolaus Kriegeskorte and Marieke Mur},
  title = {{Deep neural networks and visuo-semantic models explain complementary components of human ventral-stream representational dynamics}},
  year = {2023},
  url = {https://openreview.net/forum?id=u167dwr1Xc},
  howpublished = {OpenReview},
}
Figure