Probing Brain Activation Patterns by Dissociating Semantics and Syntax in Sentences

Shaonan Wang, Jiajun Zhang, Nan Lin, Chengqing ZongAAAI 2020 · DOI

Introduction

A long-standing debate in neuroscience concerns how our brains distinguish between the meaning of words (semantics) and the rules used to combine them (syntax). Traditionally, researchers used "hand-crafted" stimuli to study this—for example, comparing brain activity while reading a normal sentence versus a "Jabberwocky" sentence (grammatically correct but nonsensical). However, these artificial stimuli may trigger brain processes that do not occur during natural language comprehension, and results often fail to generalize. This study addresses the need for a data-driven approach to disentangle these two features from natural sentences and map them directly onto neural activation patterns.

Methods

The researchers developed the Disentangled Feature Representation Model (DFRM), a generative framework based on a Variational Autoencoder (VAE) architecture. The model aims to produce two distinct vectors for every sentence: one representing its meaning (semantic) and one representing its structure (syntactic).

Overview of the DFRM and the similarity-encoding analysis pipeline linking AI vectors to brain activations.

Model Architecture and Training

The DFRM utilizes dual encoders: a Multi-Layer Perceptron (MLP) for semantic vectors (capturing "bag-of-words" meaning) and an LSTM for syntactic vectors (capturing sequence and structure). To ensure the vectors remain distinct, the model employs targeted loss functions:

  • Paraphrase Loss (PLoss): Ensures the semantic vector remains invariant for sentences with the same meaning but different structures.
  • Jabberwocky Loss (JLoss): Ensures the syntactic vector remains invariant for sentences with identical structures but different lexical content.
  • Word Position Loss (WPLoss): Sharpens the syntactic encoder's grasp of grammar by predicting word order.

Similarity-Encoding Analysis

To link computational features to biological activity, the team performed similarity-encoding analysis. They compared the similarity of sentences in the AI-generated vector space to the similarity of brain activation patterns in specific Regions of Interest (ROIs) from a publicly available fMRI dataset (Pereira et al., 2018).

Results

The DFRM successfully disentangled semantics and syntax, achieving a 73.0% Pearson correlation on semantic similarity and a 40.0 F1 score on syntactic parsing, outperforming standard baselines like BERT and GloVe.

Neural Representation of Language

The study reveals that semantic information is processed much more robustly and widely across the brain than syntactic structure. In fMRI experiments, the model identified significantly more ROIs sensitive to semantics (112–125) than to syntax (31–48).

Similarity-encoding analysis results across various brain Regions of Interest (ROIs), showing the dominance of semantic representations.

Key findings include:

  • Functional Dissociation: While there is some overlap (18–29 regions), the brain treats these aspects as distinct streams. Several frontal and temporal regions are exclusively sensitive to meaning, while specific parts of the right superior frontal and right inferior parietal gyrus are dedicated to syntax.
  • Semantic Dominance: Semantic information is represented across extensive networks, including the default-mode, frontoparietal, and visual networks.

Discussion and Future Work

This work demonstrates that modern NLP models can serve as precise "lenses" for studying biological intelligence. However, limitations remain, such as the small fMRI sample size (5 participants) and the current focus on sentence-level analysis. Future research could extend this framework to discourse-level analysis or other cognitive features like visual and emotional representations.

Summary updated

BibTeX
@inproceedings{wang2020probing,
  author = {Shaonan Wang and Jiajun Zhang and Nan Lin and Chengqing Zong},
  title = {{Probing Brain Activation Patterns by Dissociating Semantics and Syntax in Sentences}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6457},
}
Figure