How the Brain Dynamically Constructs Sentence-Level Meanings From Word-Level Features

Nora E. Aguirre-Celis, Risto MiikkulainenFrontiers Artif. Intell 2022 · DOI

Introduction

Traditional Natural Language Processing (NLP) often treats words as static vectors. While modern contextual embeddings like BERT have improved performance, they frequently lack "symbol grounding"—the inherent understanding that a "heavy book" refers to physical mass while a "long book" refers to temporal duration. In contrast, the human brain utilizes embodied cognition, where word meanings are fluid and depend heavily on sensory and cognitive context. For instance, the word "piano" activates different neural patterns depending on whether one is playing it or moving it. This research introduces CEREBRA (Context-dEpendent meaning REpresentations in the BRAin), a model designed to quantify how the brain performs conceptual combination to construct sentence-level meaning from individual word features.

Methodology: The CEREBRA Model

The foundation of this work is Concept Attribute Representation (CAR) theory, which defines words as 66-dimensional vectors across experiential attributes such as Vision, Shape, Smell, and Emotion.

Figure 1: CAR representations for "BICYCLE" and "TABLE" illustrating the 66-dimensional experiential attributes used to define word meanings.

To simulate the dynamic shifting of these attributes, the researchers developed the CEREBRA model using the FGREP mechanism (Forming Global Representations with Extended Backpropagation). The process involves mapping static CAR vectors to "SynthWords" (approximations of fMRI brain activity), predicting sentence-level patterns, and then using backpropagation to "tweak" the original word attributes until the predicted activity matches observed human fMRI data. The resulting "Revised CARs" represent the word's meaning specifically within the context of the sentence.

Datasets and Validation

The model was trained and tested using the Glasgow Sentence Collection (240 sentences), a database of human-provided CAR ratings for 242 words, and fMRI scans from 11 healthy subjects. To validate the model's psychological relevance, a manually curated set of 77 contrasting sentences (e.g., "live mouse" vs. "dead mouse") was used to evaluate specific semantic shifts against human intuition.

Key Insights and Results

The study revealed several critical aspects of how the brain processes language:

  • Attribute Congruence: Conceptual combination is a non-linear process where overlapping attributes are mutually enhanced. For example, in "plastic bottle," physical attributes are activated, whereas in "baby bottle," the attribute of Animacy significantly alters the context.
  • Contextual Differentiation: CEREBRA successfully distinguished between different uses of the same word. For the verb "listened," it shifted attributes toward "human communication" when the subject was a person, but toward "loudness/vision" when the subject was a patient watching television.
  • Human-Model Alignment: In behavioral tests, CEREBRA’s predictions matched the direction of human-perceived changes in 77% of cases, significantly exceeding chance (68%).
  • Centrality Effect: The model captured "centrality," identifying that certain attributes become more essential to a concept in specific contexts (e.g., "smallness" for a camera).

Limitations and Future Directions

While promising, the study faced constraints due to the high cost and noise of fMRI data, limiting the dataset to 240 simple sentences. Additionally, the model is currently restricted to the 66 attributes defined by CAR theory. Future work aims to expand this "semantic alphabet" and integrate multimodal inputs, such as images, to develop a more universal theory of how the mental lexicon is organized.

Summary updated

BibTeX
@article{aguirrecelis2022how,
  author = {Nora E. Aguirre-Celis and Risto Miikkulainen},
  title = {{How the Brain Dynamically Constructs Sentence-Level Meanings From Word-Level Features}},
  journal = {Frontiers Artif. Intell.},
  year = {2022},
  doi = {10.3389/frai.2022.733163},
}
Figure