Abstract
Mostafa Abdou, Artur Kulmizev, Felix Hill, Daniel M. Low, Anders Søgaard. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
While large-scale language models like BERT and ELMo achieve state-of-the-art performance, they remain "black boxes." Traditional "probing" methods—training small classifiers to detect linguistic properties in specific layers—have significant limitations. Probing requires labeled training data, is prone to overfitting, and fails to capture the global "geometry" of a model's representational space. Crucially, it is difficult to use probing to directly compare neural network internal states with human behavioral data, such as eye-tracking metrics. This research addresses these gaps by introducing a transparent, training-free framework to compare models across different architectures and modalities.
Methodology: Representational Similarity Analysis (RSA)
The authors utilize Representational Similarity Analysis (RSA), a technique originally developed in neuroscience, to analyze model internals without additional training.
The process involves three stages:
- Representational Dissimilarity Matrices (RDMs): For a given set of sentences, the "distance" between every pair of sentences is calculated within a specific model layer, creating a matrix that defines the geometry of that layer's representational space.
- Second-order Comparison: These matrices are compared across different layers or models. A high degree of difference between RDMs for a specific sentence indicates "layer disagreement."
- Third-order Analysis: This "layer disagreement" is then correlated with external metrics, including human gaze fixation duration from eye-tracking and linguistic features like syntactic complexity (Yngve scores).
Experimental Setup
The study utilized the English portion of the Dundee Eye-tracking Corpus, which contains gaze data from native speakers reading newspaper articles. The researchers analyzed pretrained BERT-large and ELMo models. Linguistic metadata was derived from the British National Corpus (for word frequency), WordNet (for lexical ambiguity), and Yngve scores (for syntactic complexity).
Key Results: The Tripartite Affinity
The core finding of the study is that layer disagreement serves as a reliable proxy for processing difficulty. The researchers identified a strong three-way link—a "tripartite affinity"—between layer disagreement, syntactic complexity, and human fixation time.
Disagreement as a Proxy for Difficulty
Sentences that are linguistically complex or uncommon cause the internal layers of a model to "disagree" on their representation. This instability mirrors human cognition: the researchers found highly significant negative correlations (up to -0.88) between layer similarity and total fixation duration. Essentially, the less the model's layers agreed on a sentence, the longer humans spent looking at it.
Syntactic Complexity and the "Syntax Hub"
Layer disagreement showed a strong positive correlation (0.66) with Yngve scores, which measure syntactic complexity. Interestingly, this correlation was strongest in the middle layers of BERT. This provides independent, geometric evidence for the "NLP pipeline" hypothesis, suggesting that models process surface features in early layers, syntax in middle layers, and semantics in later layers.
Other Linguistic Factors
While word frequency also correlated with layer disagreement, the effect was less pronounced than syntax. Lexical ambiguity (the number of senses per word) showed almost no correlation with either layer disagreement or human fixation in this experimental setup.
Conclusion
This work demonstrates that the internal representational geometry of neural language models can serve as a mirror for human processing difficulty. By using RSA, researchers can peek inside the "black box" to find striking parallels between machine and human linguistic processing. Future work may extend this analysis to typologically diverse languages and newer architectures to see if these geometric patterns are universal across scaling and linguistic variation.
Summary updated