The time course of visuo-semantic representations in the human brain is captured by combining vision and language models

Boyan Rong, Alessandro T. Gifford, Emrah Düzel, Radoslaw Martin Cichy

Abstract

The human visual system provides us with a rich and meaningful percept of the world, transforming retinal signals into visuo-semantic representations. For a model of these representations, here we leveraged a combination of two currently dominating approaches: vision deep neural networks (DNNs) and large language models (LLMs). Using large-scale human electroencephalography (EEG) data recorded during object image viewing, we built encoding models to predict EEG responses using representations from a vision DNN, an LLM, and their fusion. We show that the fusion encoding model outperforms encoding models based on either the vision DNN or the LLM alone, as well as previous modelling approaches, in predicting neural responses to visual stimulation. The vision DNN and the LLM complemented each other in explaining stimulus-related signal in the EEG responses. The vision DNN uniquely captured earlier and broadband EEG signals, whereas the LLM uniquely captured later and low frequency signals, as well as detailed visuo-semantic stimulus information. Together, this provides a more accurate model of the time course of visuo-semantic processing in the human brain.

Introduction

When we look at an image, our brain performs a lightning-fast transformation, starting with raw retinal signals and ending with a rich, meaningful understanding of the world. While vision-based deep neural networks (DNNs) have become the standard for modeling early visual processing, they often fail to capture the high-level "meaning" or semantics of a scene. Conversely, large language models (LLMs) are masters of semantics but lack a direct connection to visual input.

This study addresses the challenge of modeling the full time course of visuo-semantic processing. By leveraging high-resolution EEG data and a novel "fusion" encoding model, researchers have mapped how the brain transitions from structural visual analysis to deep semantic interpretation.

Methodology: The Fusion Encoding Model

The researchers built encoding models—mathematical functions that predict brain activity based on features extracted from computational models. They utilized three primary approaches:

  1. Vision Model: CORnet-S, a recurrent convolutional neural network designed to mimic the hierarchical structure of the primate visual cortex.
  2. Language Model: OpenAI’s text-embedding-3-large. To bridge the gap between pixels and text, GPT-4V generated five detailed descriptions for each image, which were then converted into vector embeddings.
  3. Fusion Model: A weighted combination of both vision and language features used to predict EEG responses via ridge regression.

The study utilized the THINGS EEG2 dataset, containing recordings from 10 human participants viewing over 16,000 naturalistic images.

Overview of the encoding model pipeline, temporal prediction accuracy, and spatial unique contributions.

Key Results

The Temporal Hand-off

The analysis revealed a distinct temporal progression. Vision DNNs are the "early responders," with prediction accuracy peaking around 110ms after an image appears, primarily in the medial occipito-parietal regions. LLMs are "late bloomers," peaking much later at 365ms and spreading to bilateral occipito-temporal and parieto-frontal areas. This suggests a temporal "hand-off" where the brain first processes visual structure before engaging in complex semantic synthesis.

Spectral Fingerprints

The models also captured different neural rhythms. The vision DNN uniquely predicted broadband signals (fast, initial neural bursts), whereas the LLM uniquely predicted low-frequency signals (delta and theta bands). These low frequencies are traditionally associated with memory access and semantic integration, further supporting the LLM's role in modeling high-level cognition.

Beyond Simple Labels

A critical finding was that the LLM's predictive power stems from detailed descriptions—including context, materials, and implied narratives—rather than just simple category labels like "dog" or "hammer." As shown in the comparison of different description types, the model using full descriptions significantly outperformed those using only category labels or random text.

Comparison of fusion models using full descriptions versus simple category labels, highlighting the unique contribution of detailed semantic information.

Conclusion

By fusing vision DNNs and LLMs, this research provides a more complete and accurate model of the human visual system. The fusion model significantly outperforms individual models and established multimodal architectures like CLIP. This approach highlights that the human brain's late-stage visual processing is not just about identification, but about constructing a rich, semantic narrative of the visual world.

Summary updated

Abstract

The human visual system provides us with a rich and meaningful percept of the world, transforming retinal signals into visuo-semantic representations. For a model of these representations, here we leveraged a combination of two currently dominating approaches: vision deep neural networks (DNNs) and large language models (LLMs). Using large-scale human electroencephalography (EEG) data recorded during object image viewing, we built encoding models to predict EEG responses using representations from a vision DNN, an LLM, and their fusion. We show that the fusion encoding model outperforms encoding models based on either the vision DNN or the LLM alone, as well as previous modelling approaches, in predicting neural responses to visual stimulation. The vision DNN and the LLM complemented each other in explaining stimulus-related signal in the EEG responses. The vision DNN uniquely captured earlier and broadband EEG signals, whereas the LLM uniquely captured later and low frequency signals, as well as detailed visuo-semantic stimulus information. Together, this provides a more accurate model of the time course of visuo-semantic processing in the human brain.
BibTeX
@misc{rong2025time,
  author = {Boyan Rong and Alessandro T. Gifford and Emrah Düzel and Radoslaw Martin Cichy},
  title = {{The time course of visuo-semantic representations in the human brain is captured by combining vision and language models}},
  year = {2025},
  eprint = {2506.19497},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.19497},
  howpublished = {arXiv},
}
Figure