Experientially-grounded and distributional semantic vectors uncover dissociable representations of conceptual categories

Francesca Carota, Hamed Nili, Nikolaus Kriegeskorte, Friedemann PulvermüllerLanguage 2023 · DOI

TL;DR

This study investigates how the human brain represents word meanings, comparing two main theories: one based on word co-occurrence statistics (distributional) and another based on sensory and motor experiences (experiential). They found that while both contribute, experiential information is crucial for understanding object concepts, and different brain regions specialize in processing these distinct types of semantic information.

What Problem Does This Paper Address?

For decades, researchers have debated how humans understand word meanings. Two prominent theories have emerged:

  1. Distributional Semantics: This view suggests that word meanings are derived from the contexts in which they appear. Words that frequently co-occur with similar words are considered to have similar meanings (e.g., "dog" and "cat" often appear with "pet," "animal," "walk"). This is often captured by analyzing large text corpora.
  2. Experiential Semantics (or Grounded Cognition): This perspective argues that word meanings are grounded in our sensory and motor experiences. For example, the meaning of "peach" is linked to its roundness, color, taste, and the actions of eating it.

While both approaches have shown promise in explaining semantic representations, it's unclear how they relate to each other and how they are implemented in the brain. Specifically, do they map onto the same brain regions, or are there specialized areas for each type of information? This paper aims to disentangle these representations by comparing how well distributional and experiential models predict brain activity patterns associated with word meanings.

Methods

The researchers employed Representational Similarity Analysis (RSA), a powerful technique for comparing the similarity structure of brain activity patterns with the similarity structure predicted by computational models.

Here's a breakdown of their approach:

  • fMRI Data: They used existing fMRI data from participants who silently read action- and object-related words.
  • Semantic Models:
    • Experiential Model: This model was built using human ratings of various semantic properties for each word, such as action-relatedness, arm/leg/face-relatedness, color, shape, imageability, and body sensation.
  • Distributional Model: This model used GloVe (Global Vectors for Word Representation), a state-of-the-art word embedding technique that captures word meanings based on their co-occurrence statistics in a massive text corpus.
  • Integrative Model: This model combined the experiential and distributional models by averaging their similarity matrices.
  • RSA Procedure:
    1. For each word, they calculated the similarity (or dissimilarity) between all pairs of words based on each of the three models, creating Representational Dissimilarity Matrices (RDMs).
  • They also computed RDMs from the fMRI data, reflecting the similarity of brain activity patterns for each pair of words.
  • Finally, they correlated the model RDMs with the brain data RDMs across different brain regions (using a whole-brain searchlight approach and specific Regions of Interest - ROIs) to see which model best predicted the observed brain activity.
  • Statistical Comparisons: They used statistical tests (e.g., Wilcoxon rank sum tests, ANOVAs) to compare the performance of the different models, both for all words and for specific categories (action vs. object words).

Datasets

The study utilized an existing dataset of fMRI responses from 23 healthy volunteers. The stimuli consisted of 96 words, equally divided into six categories: leg-, arm-, and face-related actions, and tool-, animal-, and food-related objects.

The distributional model was based on pre-trained GloVe word vectors derived from a massive corpus of 42 billion tokens. The experiential model was constructed from semantic ratings collected in a separate study.

Key Insights and Ideas

  • Experiential Grounding is Key for Objects: The study strongly suggests that understanding object concepts relies heavily on their sensory and motor properties. Experiential models were significantly better at predicting brain activity for object words, particularly in visual processing areas.
  • Distributional Models Excel in Frontal Areas: Distributional models, which capture word co-occurrence patterns, showed stronger correlations with brain activity in frontal regions, especially the inferior frontal gyrus (LIFG). This suggests these areas might be involved in processing abstract semantic relationships derived from language use.
  • Category-Specific Brain Regions: The research highlights a "double dissociation" – different brain regions are more sensitive to different types of semantic information. For instance, frontal areas seem to favor distributional information for action words, while visual areas (like the fusiform and angular gyrus) are more attuned to experiential information for object words.
  • Integration Offers Broad Coverage, But Not Always Superiority: While the integrative model (combining both approaches) often showed widespread correlations across many brain regions, it wasn't always superior to the individual models in predicting specific semantic representations. This implies that sometimes, one type of information might be more dominant for a particular concept or brain region.
  • Action vs. Object Distinction: The brain clearly distinguishes between how it represents action-related words and object-related words, with different neural signatures for each.

Key Results

  • Whole-Brain RSA: The integrative model showed widespread correlations in fronto-temporal and parieto-occipital cortex. The experiential model showed similar patterns but less so in the LIFG. The distributional model was more focused in anterior frontal regions.
  • ROI Analyses:
    • For all words, the integrative model showed significant correlations in more ROIs than the other two models. The experiential model also outperformed the distributional model in several areas.
  • Action vs. Object Words: For action words, both distributional and experiential models performed well in frontal and posterior temporal areas. However, for object words, the experiential model (and the integrative model) significantly outperformed the distributional model in regions like the left fusiform gyrus (FuG) and angular gyrus (AG).
  • Model Comparisons:
    • In left pITG, integrative and experiential models showed higher correlations than the distributional model.
  • In LIFG (BA 47), the distributional model showed higher correlations.
  • For object words, the integrative and experiential models were significantly superior to the distributional model in PMTG, FuG, and AG.

Limitations and Future Work

  • Stimulus Set: The study focused on concrete written words. Future work could explore how these models apply to abstract words, sentences, or discourse.
  • Model Specificity: The performance of distributional models can vary depending on the specific algorithm and corpus used. The authors acknowledge that other models might be better optimized for explaining object concept similarity.
  • Task: The participants performed a typo-detection task during silent reading. While this avoids task-specific biases, it's a passive task. Future studies could investigate how different tasks (e.g., semantic judgments) might influence these representations.
  • Integration Nuances: While the integrative model showed broad coverage, it wasn't always superior. Further research is needed to understand the optimal ways to combine different semantic information sources and how this interplay varies across different types of concepts and linguistic contexts. The authors suggest that language-based computational models might be more valuable for predicting complex event imaginings or sentence-level meaning, while isolated word-level concepts might rely more heavily on experiential grounding.

Summary updated

Abstract

Neuronal populations code similar concepts by similar activity patterns across the human brain's semantic networks. However, it is unclear to what extent such meaning-to-symbol mapping reflects distributional statistics, or experiential information grounded in sensorimotor and emotional knowledge. We asked whether integrating distributional and experiential data better distinguished conceptual categories than each method taken separately. We examined the similarity structure of fMRI patterns elicited by visually presented action- and object-related words using representational similarity analysis (RSA). We found that the distributional and experiential/integrative models respectively mapped the high-dimensional semantic space in left inferior frontal, anterior temporal, and in left precentral, posterior inferior/middle temporal cortex. Furthermore, results from model comparisons uncovered category-specific similarity patterns, as both distributional and experiential models matched the similarity patterns for action concepts in left fronto-temporal cortex, whilst the experiential/integrative (but not distributional) models matched the similarity patterns for object concepts in left fusiform and angular gyrus.
BibTeX
@inproceedings{carota2023experientially,
  author = {Francesca Carota and Hamed Nili and Nikolaus Kriegeskorte and Friedemann Pulvermüller},
  title = {{Experientially-grounded and distributional semantic vectors uncover dissociable representations of conceptual categories}},
  booktitle = {Language},
  year = {2023},
  doi = {10.1080/23273798.2023.2232481},
}
Figure