Human Sentence Processing: Recurrence or Attention?

Danny Merkx, Stefan L. FrankNAACL 2021

TL;DR

This paper investigates whether Transformer or Recurrent Neural Network (RNN) architectures better model human sentence processing. Surprisingly, Transformers, despite their seemingly implausible "unlimited memory" mechanism, outperform RNNs in explaining human reading behavior and neural data, suggesting that attention-based retrieval might be a more accurate cognitive model than previously thought.

What Problem Does This Paper Address?

For decades, Recurrent Neural Networks (RNNs) have been the go-to architecture for modeling how humans process language, particularly in explaining reading times and brain activity (like the N400 response). RNNs process information sequentially, mirroring the step-by-step nature of reading. However, the recent advent of the Transformer architecture, which relies on "attention" mechanisms rather than strict recurrence, has revolutionized natural language processing (NLP) tasks. Despite its success in NLP, its suitability as a model for human sentence processing remained largely unknown. This paper aims to bridge this gap by directly comparing Transformer and RNN models on their ability to predict human reading effort, challenging the long-held assumption that human processing must be inherently recurrent.

Methods

The authors trained their own language models (LMs) using both GRU (a type of RNN) and Transformer architectures. This was done to ensure a fair comparison by controlling for training data, hyperparameters, and LM quality. They specifically trained single-layer and two-layer versions of both architectures. The models were trained on a large corpus of English sentences to predict the next word.

To evaluate how well these LMs model human sentence processing, the researchers used surprisal estimates derived from the LMs. Surprisal, a measure of how unexpected a word is given its context, is known to correlate with human processing difficulty. They then employed linear mixed-effects regression (LMER) models to assess the goodness-of-fit between the surprisal values from their LMs and existing human reading data (self-paced reading, eye-tracking, and EEG). Generalised additive modelling (GAM) was also used to compare the architectures at equal language modeling capability, isolating the effect of architectural differences.

Datasets

  • Training Data: The models were trained on Section 1 of the English Corpora from the Web (ENCOW 2014), specifically the 10,000 most frequent word types, resulting in approximately 5.9 million sentences and 85 million tokens.
  • Evaluation Data: The models were evaluated on previously collected human reading data from three psycholinguistic experiments:
  • Self-paced reading (SPR) data from 54 participants.
  • Eye-tracking (ET) data from 35 participants.
  • EEG data (measuring N400 amplitude) from 24 participants. The SPR stimuli consisted of 361 sentences, while the EEG and ET stimuli were subsets of the shorter SPR sentences.

Key Insights and Ideas

  • Transformers Outperform RNNs: The most striking finding is that Transformer models, despite their non-recurrent nature, provided a better fit to human reading data (SPR, EEG) than GRU models, even when controlling for language modeling quality.
  • Attention as a Cognitive Plausibility: This challenges the intuitive notion that human sentence processing must be recurrent. The Transformer's "attention" mechanism, which allows direct access to past inputs regardless of temporal distance, was previously considered cognitively implausible. However, its superior performance suggests that cue-based retrieval, potentially similar to attention, might be a more accurate cognitive model.
  • Depth Matters for Transformers, Not RNNs: Adding a second layer significantly improved the Transformer's performance, likely by enabling it to utilize implicit order information. In contrast, adding a second GRU layer did not improve performance and even slightly hurt it for reading time data.
  • Sentence Length and Architecture: The Transformer's advantage on SPR data was more pronounced for longer sentences, suggesting its "unlimited memory" might be particularly beneficial for processing longer linguistic structures.

Key Results

  • Goodness-of-Fit: Transformer models consistently showed higher goodness-of-fit scores in predicting human reading times and N400 amplitudes compared to GRU models, especially at the end of training.
  • Two-Layer Transformer Advantage: The two-layer Transformer outperformed the single-layer Transformer on the N400 data and generally achieved higher LM quality.
  • No Significant Difference on Shorter Sentences: When analyzing only shorter sentences (those present in the EEG/ET datasets), performance differences between Transformer and GRU architectures became negligible, suggesting sentence length plays a role in the observed advantage.

Limitations and Future Work

  • Cognitive Plausibility of Unlimited Memory: While Transformers perform better, the idea of "unlimited working memory" is still considered cognitively implausible. The authors suggest that cue-based retrieval theories, which incorporate recency biases, might offer a more compatible explanation for Transformer-like mechanisms in human cognition.
  • Experimental Design: The authors note that future research could resolve ambiguities by using the same set of stimuli across SPR and EEG experiments.
  • Error Analysis: More specific error analysis is needed to pinpoint which sentences or linguistic properties cause Transformers to perform better, which could lead to testable hypotheses about human sentence processing.
  • Implicit Order Information: Further investigation into how Transformers leverage implicit order information, especially with multiple layers, is warranted.

Summary updated

BibTeX
@inproceedings{merkx2021human,
  author = {Danny Merkx and Stefan L. Frank},
  title = {{Human Sentence Processing: Recurrence or Attention?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure