Visio-Linguistic Brain Encoding

Subba Reddy Oota, Jashn Arora, Vijay Rowtula, Manish Gupta, Bapi Raju Surampudi

Abstract

Recent advancements in computational neuroscience have sought to bridge the gap between artificial intelligence and human cognition through brain encoding—the prediction of neural activity from sensory stimuli. While traditional models have relied on single-mode architectures (vision-only or text-only), this paper explores the efficacy of multi-modal Transformers. We find that models integrating visual and linguistic information, specifically VisualBERT, achieve a new state-of-the-art in predicting functional Magnetic Resonance Imaging (fMRI) activity. Our results suggest that the human brain implicitly processes linguistic context even during purely visual tasks, highlighting the importance of cross-modal integration in cognitive modeling.

Introduction

The field of brain encoding aims to build computational frameworks capable of predicting how the human brain responds to specific stimuli. Historically, these models were specialized: Convolutional Neural Networks (CNNs) were used for visual stimuli, while Language Models were reserved for textual data. However, human perception is inherently multi-modal; we do not perceive the world in isolated streams but through a continuous integration of sight, sound, and language.

This study addresses two primary questions: First, how do modern Image Transformers (such as ViT) compare to traditional CNNs in their ability to map to neural activity? Second, can multi-modal Transformers, which utilize co-attention mechanisms to fuse visual and linguistic data, provide a more accurate representation of the distributed processing found in the human brain?

Methodology

To investigate these questions, researchers systematically compared three categories of architectures:

  1. Single-Mode Models: Including traditional CNNs (VGG19, ResNet50) and Image Transformers (ViT, DEiT, BEiT).
  2. Late-Fusion Models: Simple concatenations of independent visual and linguistic features.
  3. Multi-Modal Transformers: Architectures such as VisualBERT, LXMERT, and CLIP that employ deep fusion through co-attention mechanisms.

The internal representations (features) generated by these models were mapped to 3D fMRI pixels (voxels) using Ridge Regression. This mathematical mapping allows researchers to quantify how well a model's internal "understanding" of an image correlates with actual human neural responses.

Datasets

The models were evaluated against two major fMRI benchmarks:

  • BOLD5000: A large-scale dataset featuring four subjects viewing over 5,000 natural images sourced from ImageNet, COCO, and the Scene dataset.
  • Pereira: A dataset involving sixteen subjects presented with concept words paired with corresponding images (e.g., the word "dog" alongside a photograph of a dog), covering 180 distinct concepts.

Key Insights and Results

The Superiority of Multi-Modal Fusion

VisualBERT emerged as the state-of-the-art (SOTA) model, significantly outperforming both vision-only models and late-fusion multi-modal models. On the BOLD5000 dataset, VisualBERT achieved superior 2V2 Accuracy—a metric measuring the model's ability to distinguish between two stimuli based on predicted brain activity—and higher Pearson Correlation scores with actual fMRI voxels.

The "Linguistic Ghost" in Visual Processing

A pivotal finding of this research is that multi-modal models outperformed vision-only models even in brain regions traditionally categorized as purely visual. This suggests a "linguistic ghost" in the machine: when humans view an image, the brain may be implicitly generating linguistic labels or descriptions to organize the visual information. The deep fusion in models like VisualBERT appears to mimic this biological tendency more effectively than models that treat vision in isolation.

Structural and Hierarchical Alignment

The study found that the layer-wise hierarchy of Transformers mirrors the visual hierarchy of the human brain. Lower layers of the models correlate most strongly with Early Visual areas, while deeper layers align with complex processing regions such as the Lateral Occipital Complex (LOC) and the Parahippocampal Place Area (PPA). Interestingly, standard Image Transformers (ViT) often performed worse than traditional CNNs (ResNet) unless they were integrated into a multi-modal framework, suggesting that the Transformer architecture's strength in brain encoding is most apparent when handling cross-modal interactions.

Concrete vs. Abstract Concepts

Using the Pereira dataset, the researchers noted that models were significantly more accurate at predicting brain activity for concrete concepts (e.g., "hammer") than for abstract ones (e.g., "faith"). This confirms that visual grounding remains a cornerstone of current encoding models and highlights a gap in representing purely abstract thought.

Discussion and Future Work

The success of VisualBERT indicates that "co-attention"—where visual and linguistic features interact throughout the processing pipeline—is a more biologically plausible mechanism than late-stage fusion. However, limitations remain. VisualBERT is computationally intensive, and the current data relies on passive viewing tasks. Future research may find that the gap between multi-modal and single-mode models widens even further during active tasks, such as object naming or scene description.

Furthermore, while this study focused on the intersection of vision and language, the human experience is richer, involving audio and tactile feedback. The development of a "Universal Transformer" that integrates these additional modalities represents the next frontier in cracking the brain's computational code.

Summary updated

Abstract

Enabling effective brain-computer interfaces needs understanding how the human brain encodes stimuli across modalities such as visual, language (or text), etc. Brain encoding aims at constructing fMRI brain activity given a stimulus. There exist a plethora of neural encoding models which study brain encoding for single-mode stimuli: visual (pretrained CNNs) or text (pretrained language models). Few recent papers have also obtained separate visual and text representation models and performed late-fusion using simple heuristics. However, previous work has failed to explore: (a) the effectiveness of image Transformer models for encoding visual stimuli, and (b) co-attentive multi-modal modeling for visual and text reasoning. Further, as pretrained image Transformers and multi-modal Transformers have continued to evolve, it is important to understand if they are becoming more brain-like and hence lead to improved brain encoding. In this paper, we systematically explore the efficacy of image Transformers (ViT, DEiT, and BEiT) and multi-modal Transformers (VisualBERT, LXMERT, ViLBERT, and CLIP) for brain encoding. Extensive experiments on two popular datasets, BOLD5000 and Pereira, provide the following insights. (1) To the best of our knowledge, we are the first to investigate the effectiveness of image and multi-modal Transformers for brain encoding. (2) Surprisingly, we observe a better encoding correlation between Transformer model layers and the levels of visual processing in the human brain when compared to CNN architectures. (3) We find that multi-modal Transformers significantly outperform previously proposed single-mode CNNs, image Transformers as well as other previously proposed multi-modal models, thereby establishing new state-of-the-art. The supremacy of visio-linguistic models raises the question of whether the responses elicited in the visual regions are affected implicitly by linguistic processing even when passively viewing images. Future fMRI tasks can verify this computational insight in an appropriate experimental setting. We make our code publicly available.
BibTeX
@inproceedings{oota2022visio,
  author = {SUBBA REDDY OOTA and Jashn Arora and Vijay Rowtula and Manish Gupta and Bapi Raju Surampudi},
  title = {{Visio-Linguistic Brain Encoding}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=RiabgpMdOc},
}
Figure