Connecting concepts in the brain by mapping cortical representations of semantic relations

Yizhen Zhang, Kuan Han, Robert M. Worth, Zhongming Liu

TL;DR

This paper uses fMRI data from subjects listening to natural stories to build a predictive model of brain activity, revealing how the brain represents both semantic categories and the relationships between concepts. It shows that these representations are distributed across the cortex, with the default mode network playing a key role in abstracting concepts.

What Problem Does This Paper Address?

Our brains are remarkably adept at understanding and generating language, which relies on our ability to grasp the meaning of words (concepts) and how they relate to each other. While we know that a "semantic system" in the brain stores these concepts, it's been unclear how the brain connects these concepts and infers the relationships between them. Previous research often focused on isolated brain regions representing specific categories, but this paper asks a broader question: how are semantic relations themselves represented in the brain, and do these representations transcend specific conceptual domains?

Methods

The core of this research lies in developing and applying a voxel-wise encoding model. This model takes word embeddings (numerical representations of word meanings derived from a large text corpus using the word2vec algorithm) and predicts the fMRI signal in individual brain voxels.

Here's a breakdown of the key methodological steps:

  1. Word Embeddings: They used word2vec to create 300-dimensional vector representations for thousands of words. This captures semantic features learned from natural language data.
  2. Voxel-wise Encoding Model: For each voxel in the brain, they built a linear regression model that predicts the voxel's fMRI response based on the semantic features of the words being heard. This is a standard approach in neuroimaging for mapping brain activity to stimuli.
  3. Training Data: They collected extensive fMRI data (over 11 hours per subject) from participants listening to natural audio stories. This large dataset allowed them to train the encoding model robustly.
  4. Semantic Categories: They applied the trained model to predict brain responses to words from nine different semantic categories (e.g., tool, human, animal, place).
  5. Semantic Relations: They extended this to model semantic relations by using the difference vectors between word embeddings. This captures how concepts relate to each other (e.g., "finger" is part of "hand"). They then projected these relation vectors onto the cortex using the encoding model.
  6. Cross-validation and Testing: Rigorous cross-validation was used to ensure the model's predictive power. They also tested the model on an independent story not used during training to assess its generalization capabilities.
  7. Multivariate Pattern Analysis (MVPA): For semantic relations, they also employed MVPA to uncover potentially multiple underlying cortical patterns that might represent subclasses of a relation.

Datasets

  • Training Data: Hours of fMRI data from 19 human subjects listening to natural audio stories from "The Moth Radio Hour." Different stories were used for different subjects to maximize word coverage.
  • Word Embeddings: Pre-trained word2vec model trained on the Google News dataset (approx. 100 billion words).
  • Semantic Categories: 9,849 words from nine categories (tool, human, plant, animal, place, communication, emotion, change, quantity), sourced from WordNet.
  • Semantic Relations: Word pairs from the SemEval-2012 Task 2 dataset, categorized into ten types of semantic relations (e.g., whole-part, class-inclusion, object-attribute).

Key Insights and Ideas

  • Distributed, Not Segregated: Semantic categories and relations are not neatly compartmentalized into specific brain regions. Instead, they are represented by spatially overlapping and distributed cortical patterns. This challenges the idea of highly specialized "semantic centers."
  • Default Mode Network (DMN) and Abstraction: The DMN is not just a "resting state" network; it plays a crucial role in semantic processing for abstraction. As concepts become more abstract, their representation increasingly involves the DMN.
  • Attention Network and Specification: Conversely, the frontoparietal attention network (FPN) seems to be involved in conceptual specification (moving from abstract to concrete).
  • Semantic Relations as Network Dynamics: Semantic relations are encoded not by static regions but by dynamic patterns of co-occurring activation and deactivation across different brain networks. This suggests that the relationship itself is a network phenomenon.
  • Left vs. Right Hemisphere Specialization: While both hemispheres contribute to semantics, there's a subtle specialization: the left hemisphere is more selective for concrete concepts, while the right hemisphere is more selective for abstract concepts.
  • Nominal vs. Brain Representations: Human-defined semantic relations (like "similar" or "contrast") don't always map cleanly onto brain activity, suggesting that the brain's internal representation of relations might be more nuanced or structured differently than our linguistic labels. Asymmetric relations (like "whole-part") were more reliably mapped than symmetric ones.

Key Results

  • The encoding model successfully predicted fMRI responses with significant correlations across a widespread cortical network, defining a "semantic system."
  • Individual semantic categories were represented by distributed, overlapping cortical patterns, with some hemispheric lateralization observed (e.g., "tool" and "animal" more in the left, "communication" and "emotion" more in the right).
  • The "whole-part" semantic relation was consistently represented by co-activation of the DMN and deactivation of the FPN, indicating a specific neural signature for this type of conceptual progression.
  • Other semantic relations like "class-inclusion," "object-attribute," and "case relations" also showed distinct cortical patterns, often involving combinations of activation and deactivation in specific networks.
  • MVPA revealed that some relations, like "object-attribute," could be further decomposed into distinct cortical patterns representing subclasses (e.g., inanimate vs. human attributes).

Limitations and Future Work

  • Individual Differences: The study used different stories for different subjects to maximize word coverage. While they counterbalanced stories to mitigate sampling bias, individual differences in brain structure and function could still be a confounding factor. Future work could explore individual variance more deeply.
  • Symmetric Relations: The method was more effective at mapping asymmetric semantic relations (where direction matters, like "whole-part") than symmetric ones (like "similar" or "contrast"). This is because the model relies on difference vectors, which are sensitive to direction.
  • Model-Brain Alignment: While word2vec provides a powerful semantic space, it's not necessarily identical to how the brain encodes semantics. The authors acknowledge that the brain's semantic space is likely similar "up to linear projection." Future work could explore refining word embedding models based on neurobiological principles.
  • Experimental Validation: The findings are based on predictive modeling. Future studies could use these predictions to design targeted experiments to further validate the observed neural representations of concepts and relations.
  • Broader Semantic Space: The study used a large vocabulary, but the full spectrum of human concepts and their relations is vast. Expanding the scope of words and relations studied would be beneficial.

Summary updated

Abstract

In the brain, the semantic system is thought to store concepts. However, little is known about how it connects different concepts and infers semantic relations. To address this question, we collected hours of functional magnetic resonance imaging data from human subjects listening to natural stories. We developed a predictive model of the voxel-wise response and further applied it to thousands of new words. Our results suggest that both semantic categories and relations are represented by spatially overlapping cortical patterns, instead of anatomically segregated regions. Semantic relations that reflect conceptual progression from concreteness to abstractness are represented by cortical patterns of activation in the default mode network and deactivation in the frontoparietal attention network. We conclude that the human brain uses distributed networks to encode not only concepts but also relationships between concepts. In particular, the default mode network plays a central role in semantic processing for abstraction of concepts.
BibTeX
@inproceedings{zhang2020connecting,
  author = {Yizhen Zhang and Kuan Han and Robert M. Worth and Zhongming Liu},
  title = {{Connecting concepts in the brain by mapping cortical representations of semantic relations}},
  booktitle = {Nature Communications},
  year = {2020},
  doi = {10.1038/s41467-020-15804-w},
}
Figure