Decoding of generic mental representations from functional MRI data using word embeddings

Francisco Pereira, Bin Lou, Brianna Pritchett, Nancy Kanwisher, Matthew Botvinick, Evelina FedorenkobioRxiv 2016 · DOI

TL;DR

This paper introduces a novel method for building "forward models" that map semantic concepts (words, sentences) to brain activity patterns measured by fMRI. Crucially, these models can decode abstract concepts and generalize to topics not seen during training, paving the way for understanding complex thought from brain data.

What Problem Does This Paper Address?

Researchers have successfully used fMRI to decode concrete stimuli like images and videos, building "forward models" that link stimulus representations to brain activity. However, decoding abstract concepts or complex linguistic structures like sentences and paragraphs from fMRI has remained a significant challenge. This paper aims to bridge this gap by developing a method to build forward models that can handle generic, abstract, and complex conceptual stimuli, moving beyond simple concrete objects. This is important for a deeper understanding of how the brain represents and processes abstract thought and language.

Methods

The core of the approach involves two key components for building the forward model:

  • Subject-Independent Component (Figure 2A): This maps words naming concepts to their distributed semantic representations using pre-trained word embeddings (specifically GloVe). These embeddings capture semantic relationships between words in a high-dimensional space.
  • Subject-Dependent Component (Figure 2B): This maps the semantic dimensions from the word embeddings to patterns of brain activation. This is achieved by learning an "image basis" where each basis image represents the brain's response to a specific semantic dimension. The model learns a weighted combination of these basis images to reconstruct the observed fMRI pattern for a given concept.

The process involves:

  1. Stimulus Selection: A novel method for selecting a diverse set of concepts (180 in total) is proposed, aiming to "span" the semantic space. This involves clustering GloVe word embeddings and selecting representative words from each cluster.
  2. Forward Model Learning: Using fMRI data acquired while subjects engage with these concepts (presented as sentences, picture-word pairs, or word clouds), the model learns the mapping between semantic vectors and brain activation patterns.
  3. Decoding: The learned forward model is then inverted. Given new fMRI data, the model estimates a semantic vector, which is then used to identify the most likely concept from a set of candidates.

Datasets

  • fMRI Data: The study collected fMRI data from five subjects performing three tasks: attending to highlighted words in sentences, attending to picture-word pairs, and attending to words in word clouds. This data was used to train and evaluate the forward models.
  • GloVe Word Embeddings: Pre-trained 300-dimensional GloVe vectors were used as the semantic representation of words.
  • Concreteness Norms: Concreteness ratings for words were obtained from a previously published dataset [2] to investigate the relationship between concept decodability and concreteness.

Key Insights and Ideas

  • Generalizing to Abstract Concepts: The approach successfully decodes abstract concepts, demonstrating that the learned forward models are not limited to concrete objects.
  • Semantic Space Spanning is Key: The method for selecting stimuli based on how well they "span" the dimensions of the semantic space is crucial for building robust and generalizable models. This ensures that the model learns how different semantic dimensions manifest in brain activity.
  • Modality Matters, but Combination is Better: While different stimulus modalities (sentence, picture, word cloud) yield varying decoding performance, combining them (SP and SPW datasets) significantly improves decoding accuracy.
  • No Clear Link Between Concreteness and Decodability: Surprisingly, the study found no significant correlation between how concrete a concept is and how well it can be decoded from brain activity. This might be due to subjects visualizing abstract concepts or the nature of the experimental tasks.
  • Decoding Complex Stimuli: The approach shows promise for decoding semantic information from more complex stimuli like sentences and passages, extending beyond single words.

Key Results

  • Decoding Accuracy: The SPW (sentence, picture, word cloud) combined dataset achieved the highest decoding accuracy, with normalized rank accuracy reaching up to approximately 0.76 at 5% of voxels. This significantly outperforms chance performance (0.5).
  • Modality Comparison: On average, picture stimuli led to better decoding than sentence or word cloud stimuli, with sentence being marginally better than word cloud.
  • Consistency Across Subjects: Spearman correlations between decoding rank scores across subjects were high, indicating a large degree of consistency in which concepts are decodable.
  • Generalization to Sentences/Passages: While not primarily focused on rank accuracy for complex stimuli, the paper demonstrates that the forward model trained on individual concepts can be applied to estimate semantic vectors from brain activity elicited by sentences and passages, showing a similar similarity structure to text vectors.

Limitations and Future Work

  • Limited Stimuli: Due to fMRI scanning time constraints, the study was limited to 180 concepts per session. The authors acknowledge that a more comprehensive model would require a much larger vocabulary.
  • Concreteness vs. Decodability: The lack of a clear relationship between concept concreteness and decodability is noted as a surprising finding that warrants further investigation. The authors suggest that subjects might visualize abstract concepts or that the experimental design might influence this.
  • Future Directions:
    • Leveraging more stimulus modalities and larger vocabularies.
  • Developing methods for more precise reconstruction of stimuli from brain activity.
  • Extending the approach to decode sequences of related stimuli (e.g., entire stories) to constrain trajectories through semantic space.
  • Investigating the role of different brain regions and semantic categories in decoding.
  • Exploring the use of behavioral similarity judgments as ground truth for evaluating model predictions.

Summary updated

Abstract

Abstract Several different groups have demonstrated the feasibility of building forward models of functional MRI data in response to concrete stimuli such as pictures or video, and of using these models to decode or reconstruct stimuli shown while acquiring test fMRI data. In this paper, we introduce an approach for building forward models of conceptual stimuli, concrete or abstract, and for using these models to carry out decoding of semantic information from new imaging data. We show that this approach generalizes to topics not seen in training, and provides a straightforward path to decoding from more complex stimuli such as sentences or paragraphs.
BibTeX
@misc{pereira2016decoding,
  author = {Francisco Pereira and Bin Lou and Brianna Pritchett and Nancy Kanwisher and Matthew Botvinick and Evelina Fedorenko},
  title = {{Decoding of generic mental representations from functional MRI data using word embeddings}},
  year = {2016},
  doi = {10.1101/057216},
  howpublished = {bioRxiv},
}
Figure