Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains

Samira Abnar, Lisa Beinborn, Rochelle Choenni, Willem H. Zuidema

Abstract

This research introduces Representational Stability Analysis (ReStA), a novel framework designed to probe the internal dynamics of neural language models (NLMs) by measuring state changes relative to varying context lengths. By benchmarking these models against human fMRI data, the study explores the alignment between artificial and biological language processing. Findings suggest that while Transformers like BERT exhibit high hierarchical differentiation, current fMRI signals may predominantly reflect "shallow" linguistic features, as lower model layers frequently show higher correlation with brain activity than deeper, more integrated layers.

Introduction

The rapid advancement of neural language models (NLMs) has created a "black box" problem similar to that of the human brain: we understand inputs and outputs, but the internal mechanisms of information integration remain opaque. This study addresses two core challenges: understanding how different architectures (LSTMs vs. Transformers) utilize context to construct meaning and determining if these processes mirror human neurobiology. Moving beyond isolated word analysis, the authors investigate how information is combined over multiple sentences and whether current neuroimaging tools can detect "deep" linguistic processing.

Methodology

The authors utilize two primary analytical frameworks to compare models and brains:

  1. Representational Similarity Analysis (RSA): A standard neuroscientific technique that correlates similarity matrices of stimuli (e.g., word embeddings) between two systems.
  2. Representational Stability Analysis (ReStA): A novel extension that compares a model to itself while systematically varying the amount of prior context (0 to 7 sentences). This quantifies "context-dependence."

The study evaluates five models—GloVe, ELMo, GoogleLM, BERT, and the Universal Sentence Encoder (USE)—using fMRI data from the "Harry Potter" dataset (Wehbe et al., 2014).

Alignment of brain scans with word sequences across different time delays.

As shown in Figure 1, the analysis accounts for the hemodynamic response by testing various time delays between word presentation and brain scan acquisition, ensuring the temporal alignment of linguistic stimuli and neural activity.

Key Findings

Architectural Differences in Context Handling

A significant divergence exists between LSTMs and Transformers. LSTMs (ELMo, GoogleLM) tend to "saturate" their context; internal representations stabilize after approximately three sentences. In contrast, Transformers (BERT) show a continuous rate of change, indicating a higher sensitivity to long-distance dependencies.

The "Shallow" Brain Signal and Layer Differentiation

Surprisingly, the first layers of NLMs—which typically capture lexical and surface-level information—often correlate more strongly with brain activity than deeper layers responsible for global meaning integration.

Internal representational similarity matrices for GoogleLM, ELMO, and BERT, illustrating layer-wise correlations.

Figure 4 illustrates these internal dynamics. In LSTM models (GoogleLM and ELMO), the first and last layers maintain a high correlation (~0.75). However, BERT displays a vastly different profile (~0.35), reflecting its highly hierarchical structure where the last layer represents information significantly differently from the first.

Anatomical Alignment

All models aligned most effectively with the Left Anterior Temporal Lobe (LATL), a region critical for semantic and syntactic processing. However, as models become more "context-aware," they paradoxically move further away from fMRI patterns. This suggests that either NLMs integrate information in a non-human manner or fMRI resolution is insufficient to capture deep integration signals.

Discussion and Limitations

The study highlights a significant "integration gap." The low overall correlation between models and brains (generally < 0.2) underscores the difficulty of mapping silicon-based processing to carbon-based biology. A primary limitation is the coarseness of fMRI data; with a 2-second temporal resolution, it may lack the granularity to observe the rapid, deep processing occurring in the brain. Future research should move toward predictive modeling of specific voxel activations and utilize more diverse datasets to further explore these representational overlaps.

Summary updated

BibTeX
@inproceedings{abnar2019blackbox,
  author = {Samira Abnar and Lisa Beinborn and Rochelle Choenni and Willem H. Zuidema},
  title = {{Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains}},
  booktitle = {BlackboxNLP @ ACL},
  year = {2019},
  url = {https://openreview.net/forum?id=iYIYfbJwkqc},
}
Figure