Higher-order Comparisons of Sentence Encoder Representations

Mostafa Abdou, Artur Kulmizev, Felix Hill, Daniel M. Low, Anders Søgaard

Abstract

Representational Similarity Analysis (RSA) is a technique developed by neuroscientists for comparing activity patterns of different measurement modalities (e.g., fMRI, electrophysiology, behavior). As a framework, RSA has several advantages over existing approaches to interpretation of language encoders based on probing or diagnostic classification: namely, it does not require large training samples, is not prone to overfitting, and it enables a more transparent comparison between the representational geometries of different models and modalities. We demonstrate the utility of RSA by establishing a previously unknown correspondence between widely-employed pretrained language encoders and human processing difficulty via eye-tracking data, showcasing its potential in the interpretability toolbox for neural models.

Introduction

As neural language models like BERT and ELMo grow in complexity, the challenge of interpreting their internal mechanisms—the "black box" problem—becomes more acute. The prevailing method for interpreting these models is probing, which involves training a secondary classifier to detect whether a specific layer encodes linguistic properties like part-of-speech or dependency relations.

However, probing is limited by several factors: it requires significant training data, is susceptible to overfitting, and often encounters a "ceiling effect" where multiple layers achieve near-perfect accuracy, obscuring the nuances of how representations evolve across the network. This paper proposes a more transparent, training-free alternative: Representational Similarity Analysis (RSA). By applying RSA, we can quantitatively link the internal states of neural networks to human cognitive processing, specifically through eye-tracking data.

Methodology: The RSA Framework

RSA, originally borrowed from neuroscience, allows for the comparison of disparate systems (e.g., a neural network and a human brain) by focusing on the relationships between stimuli rather than the raw activations themselves. The process follows a three-step hierarchy:

  1. Representational Dissimilarity Matrices (RDMs): For a given model layer, we calculate the distance (dissimilarity) between every pair of sentences in a dataset. This creates a geometric map of that layer's representational space.
  2. Second-order Comparison: We compare these RDMs across different layers or models. If Layer A and Layer B both perceive Sentence 1 and Sentence 2 as highly similar, they are in representational agreement.
  3. Third-order Analysis: We correlate this "layer disagreement" (defined as 1 minus the correlation between layers) with external metrics of human processing difficulty.

Experimental Setup

To test this framework, we analyzed BERT-large (24 layers) and ELMo (3 layers) using the following datasets:

  • Dundee Eye-tracking Corpus: Contains gaze data from 10 native speakers reading over 2,300 sentences. We focused on Total Fixation Duration as a proxy for cognitive load.
  • British National Corpus (BNC): Used to derive word frequency metrics.
  • WordNet: Used to quantify lexical ambiguity (number of word senses).
  • Syntactic Complexity: Measured using Yngve scores to determine the structural difficulty of sentences.

Key Insights

Disagreement as a Proxy for Difficulty

The study’s most significant finding is that internal model conflict mirrors human cognitive load. When the internal layers of a model "disagree" on the representation of a sentence, human readers typically take longer to process that same sentence. This suggests that neural networks and humans share common processing bottlenecks, particularly when faced with complex or ambiguous input.

The Syntactic "Sweet Spot"

Our analysis reinforces the hypothesis that syntactic information is primarily processed in the middle layers of Transformer architectures. These middle layers showed the strongest correlation with Yngve scores, indicating that representational shifts in these layers are highly sensitive to the structural complexity of the text.

Training-Free Interpretability

Unlike probing, RSA does not require training a new model to interpret the existing one. It relies entirely on the inherent geometry of the data already present within the encoder. This makes it a more objective and efficient tool for cross-modal alignment, such as comparing digital tensor activations with biological gaze data.

Results

  • Gaze Correlation: We found a highly significant negative correlation between layer agreement and gaze duration. In BERT’s middle layers (B11–B12), the Spearman’s $\rho$ reached -0.88 for total fixation duration. Essentially, as layer disagreement increases, human gaze time increases proportionally.
  • Syntactic Sensitivity: Syntactic complexity (Yngve scores) demonstrated a strong positive correlation (0.66) with layer disagreement, particularly in the middle layers of the models.
  • Lexical Factors: While word frequency showed a moderate correlation with layer disagreement, lexical ambiguity (polysemy) had almost no effect. This suggests a divergence in how neural models and humans handle multiple word meanings compared to how they handle structural complexity.

Conclusion

By establishing a correspondence between layer disagreement and human eye-tracking metrics, this research demonstrates that RSA is a powerful addition to the interpretability toolbox. It provides a quantitative bridge between biological and artificial intelligence, showing that even without explicit supervision, neural models develop representational geometries that reflect human-like processing difficulties. Future work will explore whether these findings hold for even larger architectures, such as GPT-4, and whether RSA can be used to align language models with other modalities like fMRI brain scans.

Summary updated

Abstract

Representational Similarity Analysis (RSA) is a technique developed by neuroscientists for comparing activity patterns of different measurement modalities (e.g., fMRI, electrophysiology, behavior). As a framework, RSA has several advantages over existing approaches to interpretation of language encoders based on probing or diagnostic classification: namely, it does not require large training samples, is not prone to overfitting, and it enables a more transparent comparison between the representational geometries of different models and modalities. We demonstrate the utility of RSA by establishing a previously unknown correspondence between widely-employed pretrained language encoders and human processing difficulty via eye-tracking data, showcasing its potential in the interpretability toolbox for neural models
BibTeX
@inproceedings{abdou2019higher,
  author = {Mostafa Abdou and Artur Kulmizev and Felix Hill and Daniel M. Low and Anders Søgaard},
  title = {{Higher-order Comparisons of Sentence Encoder Representations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=vSBx5z5Ceu},
}
Figure