What aspects of NLP models and brain datasets affect brain-NLP alignment?

Subba Reddy Oota, Mariya Toneva2023 · DOI

TL;DR

This paper systematically investigates how different aspects of natural language processing (NLP) models and brain datasets influence their alignment with human brain activity. They found that text-based models, especially bidirectional ones, are significantly better at predicting brain responses than speech models, regardless of whether the input is read or heard. This work provides crucial insights for building more effective brain-inspired AI and understanding language processing in the brain.

What Problem Does This Paper Address?

Recent advances in neuroimaging, particularly with naturalistic fMRI datasets, have opened exciting avenues for understanding how the human brain processes language. Simultaneously, the development of sophisticated NLP models has led to questions about how well these models capture human language comprehension. However, the sheer variety of available NLP models (text vs. speech, unidirectional vs. bidirectional) and fMRI datasets (different stimuli, recording methods) makes it challenging to determine which factors are most critical for achieving good "brain-NLP alignment." This paper aims to systematically dissect these factors, answering questions like: Does it matter if we read or listen to stimuli? Are all fMRI datasets equally useful? How do different NLP model architectures impact their ability to predict brain activity?

Methods

The researchers employed a standard encoding model approach. This involves:

  1. Extracting representations from various NLP models for given linguistic stimuli (text or speech).
  2. Training ridge regression models to predict fMRI brain activity in specific voxels based on these NLP representations.
  3. Evaluating the prediction performance of these models.

To account for the inherent noise in fMRI data and provide a fair comparison, they also calculated the noise ceiling for each dataset. This represents the maximum achievable prediction accuracy given the data's variability. They then normalized the prediction performance of each NLP model by this noise ceiling to get a "normalized predictivity" score.

Datasets

The study utilized five publicly available naturalistic fMRI datasets:

  • Reading: Harry-Potter (Wehbe et al., 2014)
  • Listening: Moth-Radio-Hour (LeBel et al., 2022), Narratives (Nastase et al., 2021)
  • Both Reading and Listening (for comparison): Deniz et al. (2019) dataset

Key Insights and Ideas

  • Text Dominance: Text-based NLP models consistently outperform speech-based models in predicting brain activity, even when the brain data comes from listening to speech. This suggests that the abstract linguistic representations learned by text models are more fundamental to language comprehension than the acoustic features captured by speech models.
  • Bidirectional is Better: Bidirectional transformer models (like BERT) generally achieve higher brain alignment than unidirectional models (like GPT-2). This highlights the importance of considering context from both past and future words for understanding language, mirroring how the brain likely processes information.
  • Modality Matters (but less than model type): While there are regional differences in how well text and speech models predict brain activity (e.g., similar performance in the auditory cortex, but text models excelling elsewhere), the type of NLP model (text vs. speech, bidirectional vs. unidirectional) has a more significant impact on overall alignment than the stimulus modality (reading vs. listening).
  • Dataset Variability: The estimated "noise ceiling" (a measure of data quality and consistency) can vary across datasets, influencing how well any model can be expected to predict brain activity.

Key Results

  • Text vs. Speech: All text-based NLP models showed statistically significant higher normalized brain predictivity compared to all speech models, across both reading and listening modalities.
  • Bidirectional vs. Unidirectional: BERT (bidirectional) generally outperformed GPT-2 (unidirectional) in predicting brain alignment across most datasets, though not always statistically significant.
  • Regional Differences: Speech and text models showed similar prediction performance in the auditory cortex for listening tasks. However, text models significantly outperformed speech models in other language-dominant brain regions.
  • Noise Ceiling: The average noise ceiling was similar for reading and listening of the same stimuli, suggesting that the inherent variability in brain responses doesn't drastically differ between these modalities for the same content.

Limitations and Future Work

  • Model Architectures: The study focused on a subset of popular Transformer models. Future work could explore a wider range of architectures, including more recent or specialized models.
  • Dataset Specificity: While the study used multiple datasets, further investigation into how specific dataset characteristics (e.g., stimulus complexity, participant demographics, recording quality) influence brain-NLP alignment is warranted.
  • Regional Analysis: While regional differences were noted, more in-depth analysis is needed to pinpoint which specific brain regions are most sensitive to different NLP model features and how these align with specialized language processing areas.
  • Future Research: The authors suggest that future research should focus on identifying which brain regions are most crucial for different aspects of speech and language processing and how various neural models can better capture neural activity in these regions. This could lead to more targeted model development and a deeper understanding of the neural basis of language.

Summary updated

Abstract

International audience
BibTeX
@misc{oota2023what,
  author = {Subba Reddy Oota and Mariya Toneva},
  title = {{What aspects of NLP models and brain datasets affect brain-NLP alignment?}},
  year = {2023},
  doi = {10.32470/ccn.2023.1273-0},
}
Figure