Modulation of decodable semantic features of brain activity via selection attention

Akihiro Okuno, Hiroto Yamaguchi, Shinji Nishimoto, Tomoya NakaibioRxiv 2025 · DOI

TL;DR

This study investigates how selective attention influences our ability to decode semantic information from brain activity when processing language across different modalities (text and speech). They found that while semantic features are decodable and largely shared across modalities, selective attention significantly enhances this decodability, particularly for attended stimuli, suggesting attention sharpens semantic representations.

What Problem Does This Paper Address?

We constantly process information from various sources, like reading text and listening to speech simultaneously. To make sense of this, we often need to focus our attention on one modality over another. While we know attention changes brain activity, it's been unclear which aspects of linguistic meaning are affected and how this impacts our ability to "read" semantic content directly from brain signals. This paper aims to bridge this gap by exploring how selective attention modulates the decodability of semantic features from fMRI data, and whether these decodable features are consistent across visual (text) and auditory (speech) modalities.

Methods

The researchers employed a decoding approach using fMRI data. They trained models to predict semantic features (represented as high-dimensional vectors derived from Wikipedia2Vec) from brain activity.

  • Unimodal Training: Models were trained on brain activity from either speech-only or text-only conditions.
  • Cross-modal Decoding: They then tested how well these unimodal models could decode semantic features from brain activity in the other modality.
  • Bimodal Experiment: Participants were presented with simultaneous speech and text, but instructed to attend to only one. Decoding models trained on unimodal data were then applied to this bimodal data to assess the impact of selective attention on attended vs. ignored stimuli.
  • Feature Representation: Semantic features were extracted using Wikipedia2Vec, which embeds words and entities into vectors based on their context and link information within Wikipedia. This provides a rich, distributed representation of meaning.
  • Decoding Accuracy: Measured using Pearson's correlation between predicted and actual semantic features.
  • Brain Region Analysis: Decoder weights were analyzed to identify brain regions contributing to semantic decoding.

Datasets

The study utilized fMRI data from a previous experiment (Nakai, Yamaguchi, and Nishimoto 2021) involving six healthy participants. The data comprised naturalistic narrative stories presented in unimodal (speech-only, text-only) and bimodal (simultaneous speech and text, with selective attention) conditions. No new datasets were introduced.

Key Insights and Ideas

  • Modality-Invariant Semantics: Semantic information can be decoded from brain activity regardless of whether the input is visual text or auditory speech. This suggests a shared neural representation for meaning across modalities.
  • Attention as a Sharpening Tool: Selective attention doesn't just boost overall brain activity; it specifically enhances the decodability of semantic features, making them clearer and more distinct in brain signals.
  • Attended vs. Ignored: When processing simultaneous inputs, brain activity related to the attended modality yields more decodable semantic information than that of the ignored modality.
  • Consistency Under Attention: Decodable semantic features are more consistent across modalities when attention is directed towards the linguistic content. Without attention, this cross-modal consistency diminishes.

Key Results

  • Unimodal Decoding: High decoding accuracies were achieved for semantic features in both speech-only and text-only conditions.
  • Cross-modal Decoding: Decodable features were also consistent across modalities in cross-modal decoding (e.g., training on speech, testing on text), indicating shared semantic representations.
  • Bimodal Performance: In bimodal conditions, decoding accuracy was significantly higher for attended stimuli compared to ignored stimuli, especially when training and test modalities matched.
  • Feature Consistency: Decodable features were more consistent across modalities for attended stimuli than for ignored stimuli.
  • Brain Regions: Bilateral occipital regions were important for visual decoding, while the bilateral superior temporal lobe was crucial for decoding in both modalities. Regions around the Sylvian fissure showed high correlations in model weights across modalities, suggesting shared processing.

Limitations and Future Work

  • Latent Features Only: The study decoded latent semantic features, not specific words (like nouns or verbs). It's unclear if modality and attention effects extend to decoding specific word types. Future work could investigate this.
  • Semantic Focus: The study focused solely on semantic information using Wikipedia2Vec. It would be valuable to explore if similar findings hold for other linguistic aspects like phonology, syntax, or context.
  • Cumulative Effects: The interaction between modality matching and selective attention was significant. Future research should explore how these processes cumulatively affect brain regions like the perisylvian cortex.

Summary updated

Abstract

Abstract We frequently encounter linguistic information in multiple modalities, such as text and speech, simultaneously. In such cases, we understand the information by selectively attending to one of the modalities. Previous research has shown that selective attention to a specific stimulus modality modulates cortical activity patterns. However, it remains unclear which aspects of linguistic information are selectively modulated by attention and how such modulation influences the decodability of semantic content. To address this question, we constructed decoding models of latent semantic features from the functional magnetic resonance imaging data of six participants in both unimodal (either visual or auditory) and bimodal conditions (simultaneously visual and auditory, with participants attending to a single modality at a time). In unimodal conditions, we successfully decoded the semantic contents from the brain activity. Decodable features were consistent across modalities in both intra-modal and cross-modal decoding. For bimodal conditions, decoding accuracies for the attended stimuli were higher than for the ignored when training and test stimuli belonged to the same modality. Furthermore, decodable features were more consistent across modalities with attended than ignored stimuli in both intra-modal and cross-modal decoding. These results indicate common decodable semantic features regardless of the presentation modality and that selective attention enhances the semantic representations contributing to such decodability.
BibTeX
@misc{okuno2025modulation,
  author = {Akihiro Okuno and Hiroto Yamaguchi and Shinji Nishimoto and Tomoya Nakai},
  title = {{Modulation of decodable semantic features of brain activity via selection attention}},
  year = {2025},
  doi = {10.1101/2025.06.13.659505},
  howpublished = {bioRxiv},
}
Figure