Abstract
Humans possess the ability to manipulate conceptual information independently of the sensory modality through which it was received. This study introduces "modality-agnostic" fMRI decoders that leverage modality-invariant brain representations to predict stimulus content across vision, language, and mental imagery. Utilizing the new large-scale SemReps-8K dataset—comprising fMRI responses to images, text descriptions, and mental imagery—we demonstrate that modality-agnostic decoders match or exceed the performance of modality-specific models. Searchlight analyses further reveal a widespread, left-lateralized cortical network supporting these invariant representations, which are particularly effective for decoding purely internal mental imagery.
Introduction
A central goal in cognitive neuroscience is decoding mental states from brain activity. While traditional decoders are often specialized for a single modality (e.g., visual or linguistic), human cognition is fundamentally modality-invariant; the concept of an object remains consistent whether seen, read about, or imagined. Previous research has been limited by the lack of large-scale, matched datasets across modalities. This paper addresses this gap by developing a modality-agnostic decoding framework and introducing the SemReps-8K dataset to train and validate these models.
Methods
The core of our approach is the training of a single linear ridge regression model on fMRI responses evoked by both visual (images) and linguistic (text) stimuli.
Feature Extraction and Mapping
We utilized latent representations from state-of-the-art AI models—including Dino-v2 (vision), GPT-2/Llama2 (language), and CLIP/ImageBind (multimodal)—as targets for brain decoding. fMRI voxel activity was mapped to these high-level feature spaces.
Searchlight Analysis and Mental Imagery
To localize modality-invariant representations, we performed a searchlight analysis across the cortex, evaluating the decoder's performance at every location. We also tested the generalization of decoders trained on external stimuli (vision and text) to internal stimuli (mental imagery), where subjects projected visual scenes based on memorized captions.
The SemReps-8K Dataset
The study introduces SemReps-8K, a comprehensive resource featuring:
- Scale: 6 subjects across 10–11 scanning sessions.
- Stimuli: Over 8,000 unique image-caption pairs from the MS COCO dataset.
- Conditions: Matched trials for viewing images, reading captions, and performing mental imagery.
Results
Performance of Modality-Agnostic Decoders
Modality-agnostic decoders achieved a mean pairwise accuracy of 81.86%, significantly outperforming models trained exclusively on images (78.15%) or captions (72.52%). Notably, these decoders were superior when predicting the content of mental imagery, reaching approximately 72.5% accuracy in identifying specific imagined scenes.
Cortical Distribution of Invariance
Searchlight mapping revealed that modality-invariant representations are not confined to small "semantic hubs" but are distributed across a broad, left-lateralized network. Key regions include the left supramarginal gyrus, inferior parietal cortex, and posterior superior temporal sulcus.
Discussion
Our findings suggest that training decoders on multiple modalities simultaneously captures a more robust, "conceptual" representation of information. The success of these models in decoding mental imagery indicates that imagination relies on the same modality-invariant cortical hubs used for integrating vision and language. Furthermore, the effectiveness of Large Language Model (LLM) features in brain decoding suggests that these AI models capture structural conceptual properties similar to those in the human brain.
Limitations
The "one-back" task used during scanning may have encouraged cross-modal processing, potentially increasing observed invariance. Additionally, signal drop-off in the anterior temporal lobes may have masked semantic information in those regions. Future work should incorporate additional modalities like audio and explore more complex hierarchical encoding models.
Summary updated