Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)

Mariya Toneva, Leila Wehbe

TL;DR

By mapping internal representations from modern NLP models to human brain recordings (fMRI and MEG), the authors reveal how different neural architectures encode context and linguistic hierarchy across layers. Crucially, modifying BERT's shallow layers with uniform attention—a tweak inspired by what best predicted brain activity—significantly improves the model’s performance on syntactic evaluation benchmarks, demonstrating that brain-alignment can directly benefit machine language understanding.

What Problem Does This Paper Address?

While large neural language models excel across language tasks, they do so without explicitly programmed rules, making their internal representations notoriously opaque. Standard probing benchmarks test for isolated linguistic features (e.g., part-of-speech or tree depth), but they struggle to evaluate complex, naturalistic context integration and high-level semantics.

Instead of relying solely on artificial probe tasks, this paper leverages the only proven system that comprehends natural language: the human brain. The authors treat brain activity as a "demultiplexer" that decomposes high-dimensional network representations across neuroanatomical regions and time latencies with established cognitive roles.

Methods

The authors investigate four representative NLP models: ELMo (recurrent LSTM), Universal Sentence Encoder (USE) (deep averaging network), BERT (bidirectional transformer), and Transformer-XL (T-XL) (transformer with segment-level recurrence).

  • Ridge Regression Encoding Models: The authors extract intermediate hidden representations $x_{\ell, k}$ (at layer $\ell$ with context window $k$) and train linear ridge regression models to predict brain activity (voxel-level fMRI or sensor/time MEG) across subjects reading identical text.
  • Classification Metric: Predictions are evaluated via a 2-versus-2 classification task on held-out test folds, measuring how accurately predicted brain activity matches true recordings.
  • Data-Driven Brain Priors: Rather than assuming unproven cognitive theories, they group brain regions based on empirical fMRI responses to narrative stimuli (Lerner et al., 2011): Group 1 areas process both isolated words and word sequences, whereas Group 2 areas respond selectively to extended, multi-word contexts.
  • Attention Ablation & Transfer to NLP: In BERT, the authors replace learned self-attention heads with uniform attention over previous-layer representations and evaluate the resulting models on 13 targeted subject-verb agreement tasks (Marvin & Linzen, 2018) via Goldberg’s masked language model probe framework.

Datasets

  • Neuroimaging Data:
  • fMRI: 8 subjects reading Chapter 9 of Harry Potter and the Sorcerer's Stone (one word every 0.5s; 45 minutes; TR = 2s) from Wehbe et al. (2014b).
  • MEG: 3 subjects reading the same chapter under identical presentation timing (Wehbe et al., 2014a).
  • NLP Probing Benchmark: 13 targeted syntactic agreement sub-tasks from Marvin & Linzen (2018), evaluating subject-verb agreement and reflexive anaphora across complex syntactic structures (e.g., prepositional phrases, relative clauses).

Key Insights and Ideas

  • Middle Layers Dominate Context Processing: For context lengths beyond 15 words, the middle layers of transformers (layer 7 in BERT, layer 11 in T-XL) predict brain activity significantly better than initial or final layers.
  • Transformer-XL Retains Long-Range Dependencies: While context representation accuracy in ELMo, BERT, and USE degrades or saturates as context expands beyond 15–20 words, T-XL’s performance steadily increases with longer contexts, validating its recurrent memory design.
  • Shallow Layers Prefer Uniform Attention: Deep layers in BERT rely heavily on structured attention to predict brain activity. Surprisingly, replacing learned attention with simple uniform averaging in shallow layers (layers 1–6) improves brain prediction accuracy for context lengths up to 25 words.
  • Closing the Brain–NLP Loop: Making early attention uniform in BERT not only aligns it better with human brain responses but also improves its zero-shot syntactic accuracy on targeted linguistic tasks.

Key Results

  • Syntactic Probe Improvements: Applying uniform attention to individual shallow layers (L1, L2, L6) in BERT yielded statistically significant gains over base BERT in 8 of the 13 Marvin & Linzen agreement tasks, while matching performance in 4 of the remaining 5 (paired $t$-test, $p < 0.01$, FDR controlled). For example:
  • Long VP coordination: jumps from 0.98 (base) to 1.00 (uniform L6).
  • Across a prepositional phrase: increases from 0.85 (base) to 0.93 (uniform L2).
  • Reflexive anaphora (simple): increases from 0.94 (base) to 0.99 (uniform L6).
  • Negative Control in Deep Layers: Layer 11 suffered the sharpest drop in brain-prediction accuracy when forced to use uniform attention; accordingly, uniform attention at layer 11 degraded or tied base performance in 12 of 13 syntactic tasks.

Limitations and Future Work

  • Risk of Reverse Inference: Inferring cognitive mechanisms from localized brain activations is prone to error; the authors mitigate this by relying strictly on empirical naturalistic priors and cross-validated predictive accuracy.
  • Stimulus Scope: The neuroimaging recordings are currently limited to a single reading dataset (one chapter of Harry Potter).
  • Future Directions: Expanding high-temporal-resolution MEG analyses to dissect fine-grained word-embedding components, constructing richer semantic brain priors for fMRI, and exploring broader architectural modifications directly inspired by neural language pathways.

Summary updated

Abstract

Neural networks models for NLP are typically implemented without the explicit encoding of language rules and yet they are able to break one performance record after another. This has generated a lot of research interest in interpreting the representations learned by these networks. We propose here a novel interpretation approach that relies on the only processing system we have that does understand language: the human brain. We use brain imaging recordings of subjects reading complex natural text to interpret word and sequence embeddings from 4 recent NLP models - ELMo, USE, BERT and Transformer-XL. We study how their representations differ across layer depth, context length, and attention type. Our results reveal differences in the context-related representations across these models. Further, in the transformer models, we find an interaction between layer depth and context length, and between layer depth and attention type. We finally hypothesize that altering BERT to better align with brain recordings would enable it to also better understand language. Probing the altered BERT using syntactic NLP tasks reveals that the model with increased brain-alignment outperforms the original model. Cognitive neuroscientists have already begun using NLP networks to study the brain, and this work closes the loop to allow the interaction between NLP and cognitive neuroscience to be a true cross-pollination.
BibTeX
@inproceedings{toneva2019interpreting,
  author = {Mariya Toneva and Leila Wehbe},
  title = {{Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=q_-RUYw5CFK},
}

Loading related content...

Figure