Author response: Modality-Agnostic Decoding of Vision and Language from fMRI

Mitja Nikolaus, Milad Mozafari, I. Berry, Nicholas Asher, Leila Reddy, Rufin VanRullenOpenalex Unknown 2025 · DOI · OpenAlex

Abstract

Humans possess the ability to manipulate conceptual information independently of the sensory modality through which it was received. This study introduces "modality-agnostic" fMRI decoders that leverage modality-invariant brain representations to predict stimulus content across vision, language, and mental imagery. Utilizing the new large-scale SemReps-8K dataset—comprising fMRI responses to images, text descriptions, and mental imagery—we demonstrate that modality-agnostic decoders match or exceed the performance of modality-specific models. Searchlight analyses further reveal a widespread, left-lateralized cortical network supporting these invariant representations, which are particularly effective for decoding purely internal mental imagery.

Introduction

A central goal in cognitive neuroscience is decoding mental states from brain activity. While traditional decoders are often specialized for a single modality (e.g., visual or linguistic), human cognition is fundamentally modality-invariant; the concept of an object remains consistent whether seen, read about, or imagined. Previous research has been limited by the lack of large-scale, matched datasets across modalities. This paper addresses this gap by developing a modality-agnostic decoding framework and introducing the SemReps-8K dataset to train and validate these models.

Figure 1: Overview of the modality-agnostic decoding framework and the SemReps-8K dataset structure.

Methods

The core of our approach is the training of a single linear ridge regression model on fMRI responses evoked by both visual (images) and linguistic (text) stimuli.

Feature Extraction and Mapping

We utilized latent representations from state-of-the-art AI models—including Dino-v2 (vision), GPT-2/Llama2 (language), and CLIP/ImageBind (multimodal)—as targets for brain decoding. fMRI voxel activity was mapped to these high-level feature spaces.

Searchlight Analysis and Mental Imagery

To localize modality-invariant representations, we performed a searchlight analysis across the cortex, evaluating the decoder's performance at every location. We also tested the generalization of decoders trained on external stimuli (vision and text) to internal stimuli (mental imagery), where subjects projected visual scenes based on memorized captions.

The SemReps-8K Dataset

The study introduces SemReps-8K, a comprehensive resource featuring:

  • Scale: 6 subjects across 10–11 scanning sessions.
  • Stimuli: Over 8,000 unique image-caption pairs from the MS COCO dataset.
  • Conditions: Matched trials for viewing images, reading captions, and performing mental imagery.

Results

Performance of Modality-Agnostic Decoders

Modality-agnostic decoders achieved a mean pairwise accuracy of 81.86%, significantly outperforming models trained exclusively on images (78.15%) or captions (72.52%). Notably, these decoders were superior when predicting the content of mental imagery, reaching approximately 72.5% accuracy in identifying specific imagined scenes.

Cortical Distribution of Invariance

Searchlight mapping revealed that modality-invariant representations are not confined to small "semantic hubs" but are distributed across a broad, left-lateralized network. Key regions include the left supramarginal gyrus, inferior parietal cortex, and posterior superior temporal sulcus.

Figure 4: Searchlight analysis results highlighting brain regions with high modality-invariant decoding performance.

Discussion

Our findings suggest that training decoders on multiple modalities simultaneously captures a more robust, "conceptual" representation of information. The success of these models in decoding mental imagery indicates that imagination relies on the same modality-invariant cortical hubs used for integrating vision and language. Furthermore, the effectiveness of Large Language Model (LLM) features in brain decoding suggests that these AI models capture structural conceptual properties similar to those in the human brain.

Limitations

The "one-back" task used during scanning may have encouraged cross-modal processing, potentially increasing observed invariance. Additionally, signal drop-off in the anterior temporal lobes may have masked semantic information in those regions. Future work should incorporate additional modalities like audio and explore more complex hierarchical encoding models.

Summary updated

Abstract

Humans perform tasks involving the manipulation of inputs regardless of how these signals are perceived by the brain, thanks to representations that are invariant to the stimulus modality. In this paper, we present modality-agnostic decoders that leverage such modality-invariant representations to predict which stimulus a subject is seeing, irrespective of the modality in which the stimulus is presented. Training these modality-agnostic decoders is made possible thanks to our new large-scale fMRI dataset SemReps-8K, released publicly along with this paper. It comprises 6 subjects watching both images and short text descriptions of such images, as well as conditions during which the subjects were imagining visual scenes. We find that modality-agnostic decoders can perform as well as modality-specific decoders, and even outperform them when decoding captions and mental imagery. Further, a searchlight analysis revealed that large areas of the brain contain modality-invariant representations. Such areas are also particularly suitable for decoding visual scenes from the mental imagery condition.
BibTeX
@inproceedings{nikolaus2025author,
  author = {Mitja Nikolaus and Milad Mozafari and I. Berry and Nicholas Asher and Leila Reddy and Rufin VanRullen},
  title = {{Author response: Modality-Agnostic Decoding of Vision and Language from fMRI}},
  booktitle = {Openalex Unknown},
  year = {2025},
  doi = {10.7554/elife.107933.2.sa0},
}
Figure