Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)

Mariya Toneva, Leila Wehbe

Abstract

State-of-the-art natural language processing (NLP) models, while effective, often operate as "black boxes" with opaque internal representations. This paper proposes a novel framework that uses human brain activity as a gold standard for language comprehension to interpret and improve these models. By aligning the internal embeddings of models like BERT and Transformer-XL with fMRI and MEG recordings of humans reading natural text, the authors identify layers that capture specific linguistic functions. Furthermore, they demonstrate that modifying model architectures to better align with brain-like processing—specifically using uniform attention in early layers—significantly improves performance on complex syntactic tasks.

Introduction

The rapid advancement of deep learning has led to NLP models that excel at a wide range of tasks. However, understanding how these models represent language remains a challenge. Traditional probing methods rely on artificial tasks, which may not reflect true comprehension. This research bridges cognitive neuroscience and AI by leveraging the human brain's proven capacity for language understanding to evaluate and refine machine representations.

Problem Statement

Despite their success, deep NLP models lack transparency. We do not fully understand the nuances of their internal embeddings. The authors argue that if a model's representations can accurately predict brain activity in regions associated with specific linguistic functions, we can gain insight into what information the model is processing.

Methodology: Aligning Models to Brain Activity

The researchers developed an alignment approach to map NLP model representations to human brain activity. This process involves acquiring brain data (fMRI/MEG) and model features for the same natural text, then training encoding models to predict brain activity from model layers.

The four-step process for aligning NLP model representations with brain activity: acquiring data, extracting features, training encoding models, and evaluating predictive power.
  1. Data Acquisition: fMRI and MEG recordings were collected from subjects reading Chapter 9 of Harry Potter and the Sorcerer’s Stone.
  2. Feature Extraction: The same text was processed through ELMo, BERT, USE, and Transformer-XL, with representations extracted from every layer.
  3. Encoding Models: A linear regression model was trained to map model layer representations to brain activity.
  4. Significance Testing: A classification task determined if the model was predictive of brain activity, identifying regions predicted by specific layers.

Interpreting Representations with Brain Priors

To interpret the results, the authors used established neuroscience "priors"—specific brain regions known to handle either isolated words (Group 1) or long-range context (Group 2). By seeing which layers predict activity in which regions, they can "decompose" complex embeddings into constituent linguistic parts.

Brain regions used as priors to interpret whether model layers represent word-level features or long-range context.

Key Insights and Results

  • The Middle Layer Peak: In transformer models, middle layers (e.g., BERT Layer 7) are the most "brain-like," balancing contextual integration with generalizability.
  • Transformer-XL and Long-Range Context: Transformer-XL uniquely maintains brain-prediction performance as context length increases, validating its recurrence mechanism for long-range dependencies.
  • Uniform Attention in Early Layers: BERT’s early layers were found to be "over-specialized." Replacing learned attention with uniform attention in layers 1-6 made the model more brain-aligned and improved syntactic accuracy in 8 out of 13 tasks.
  • Temporal Dynamics: MEG analysis confirmed that ELMo embeddings capture word-length information at 100ms and part-of-speech information at 200ms, mirroring the temporal progression of human brain activity.

Conclusion

This work demonstrates that the human brain can serve as a powerful tool for interpreting and improving AI. By aligning machine representations with biological ones, researchers can identify architectural inefficiencies and develop more robust, "brain-like" language models. Future research aims to integrate brain data directly into the training process of NLP models.

Summary updated

BibTeX
@inproceedings{toneva2019interpreting,
  author = {Mariya Toneva and Leila Wehbe},
  title = {{Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=q_-RUYw5CFK},
}
Figure