Abstract
Recent advancements in computational neuroscience have sought to bridge the gap between artificial intelligence and human cognition through brain encoding—the prediction of neural activity from sensory stimuli. While traditional models have relied on single-mode architectures (vision-only or text-only), this paper explores the efficacy of multi-modal Transformers. We find that models integrating visual and linguistic information, specifically VisualBERT, achieve a new state-of-the-art in predicting functional Magnetic Resonance Imaging (fMRI) activity. Our results suggest that the human brain implicitly processes linguistic context even during purely visual tasks, highlighting the importance of cross-modal integration in cognitive modeling.
Introduction
The field of brain encoding aims to build computational frameworks capable of predicting how the human brain responds to specific stimuli. Historically, these models were specialized: Convolutional Neural Networks (CNNs) were used for visual stimuli, while Language Models were reserved for textual data. However, human perception is inherently multi-modal; we do not perceive the world in isolated streams but through a continuous integration of sight, sound, and language.
This study addresses two primary questions: First, how do modern Image Transformers (such as ViT) compare to traditional CNNs in their ability to map to neural activity? Second, can multi-modal Transformers, which utilize co-attention mechanisms to fuse visual and linguistic data, provide a more accurate representation of the distributed processing found in the human brain?
Methodology
To investigate these questions, researchers systematically compared three categories of architectures:
- Single-Mode Models: Including traditional CNNs (VGG19, ResNet50) and Image Transformers (ViT, DEiT, BEiT).
- Late-Fusion Models: Simple concatenations of independent visual and linguistic features.
- Multi-Modal Transformers: Architectures such as VisualBERT, LXMERT, and CLIP that employ deep fusion through co-attention mechanisms.
The internal representations (features) generated by these models were mapped to 3D fMRI pixels (voxels) using Ridge Regression. This mathematical mapping allows researchers to quantify how well a model's internal "understanding" of an image correlates with actual human neural responses.
Datasets
The models were evaluated against two major fMRI benchmarks:
- BOLD5000: A large-scale dataset featuring four subjects viewing over 5,000 natural images sourced from ImageNet, COCO, and the Scene dataset.
- Pereira: A dataset involving sixteen subjects presented with concept words paired with corresponding images (e.g., the word "dog" alongside a photograph of a dog), covering 180 distinct concepts.
Key Insights and Results
The Superiority of Multi-Modal Fusion
VisualBERT emerged as the state-of-the-art (SOTA) model, significantly outperforming both vision-only models and late-fusion multi-modal models. On the BOLD5000 dataset, VisualBERT achieved superior 2V2 Accuracy—a metric measuring the model's ability to distinguish between two stimuli based on predicted brain activity—and higher Pearson Correlation scores with actual fMRI voxels.
The "Linguistic Ghost" in Visual Processing
A pivotal finding of this research is that multi-modal models outperformed vision-only models even in brain regions traditionally categorized as purely visual. This suggests a "linguistic ghost" in the machine: when humans view an image, the brain may be implicitly generating linguistic labels or descriptions to organize the visual information. The deep fusion in models like VisualBERT appears to mimic this biological tendency more effectively than models that treat vision in isolation.
Structural and Hierarchical Alignment
The study found that the layer-wise hierarchy of Transformers mirrors the visual hierarchy of the human brain. Lower layers of the models correlate most strongly with Early Visual areas, while deeper layers align with complex processing regions such as the Lateral Occipital Complex (LOC) and the Parahippocampal Place Area (PPA). Interestingly, standard Image Transformers (ViT) often performed worse than traditional CNNs (ResNet) unless they were integrated into a multi-modal framework, suggesting that the Transformer architecture's strength in brain encoding is most apparent when handling cross-modal interactions.
Concrete vs. Abstract Concepts
Using the Pereira dataset, the researchers noted that models were significantly more accurate at predicting brain activity for concrete concepts (e.g., "hammer") than for abstract ones (e.g., "faith"). This confirms that visual grounding remains a cornerstone of current encoding models and highlights a gap in representing purely abstract thought.
Discussion and Future Work
The success of VisualBERT indicates that "co-attention"—where visual and linguistic features interact throughout the processing pipeline—is a more biologically plausible mechanism than late-stage fusion. However, limitations remain. VisualBERT is computationally intensive, and the current data relies on passive viewing tasks. Future research may find that the gap between multi-modal and single-mode models widens even further during active tasks, such as object naming or scene description.
Furthermore, while this study focused on the intersection of vision and language, the human experience is richer, involving audio and tactile feedback. The development of a "Universal Transformer" that integrates these additional modalities represents the next frontier in cracking the brain's computational code.
Summary updated