So Cloze Yet So Far: N400 Amplitude Is Better Predicted by Distributional Information Than Human Predictability Judgements

James A. Michaelov, Seana Coulson, Benjamin K. Bergen

TL;DR

This study reveals that advanced language models (LMs) like GPT-3, RoBERTa, and ALBERT are better at predicting the N400 brain response to words than traditional human-based "cloze probability" measures. This suggests that the brain's processing of language might be more sensitive to statistical patterns in language than previously assumed, challenging the long-held reliance on human judgments for understanding language prediction.

What Problem Does This Paper Address?

Understanding how humans predict upcoming words during language comprehension is a fundamental question in psycholinguistics. The N400, a component of the event-related brain potential (ERP), is a well-established neural marker of processing difficulty, and its amplitude is known to decrease for predictable words. For decades, "cloze probability" – the percentage of people who would fill a blank in a sentence with a specific word – has been the gold standard for measuring this predictability. However, cloze probability has limitations: it's an aggregate measure, can be difficult to obtain for large datasets, and might not fully capture the nuances of online prediction. This paper investigates whether modern computational language models, which learn statistical patterns directly from vast amounts of text, can provide a more accurate and insightful measure of word predictability as reflected in the N400 response.

Methods

The researchers utilized existing EEG data from a large-scale experiment [33] where participants read sentences word-by-word, and ERPs were time-locked to target words. They compared the predictive power of three types of measures for the N400 amplitude:

  1. Cloze Probability: The traditional measure derived from human participants filling in sentence gaps.
  2. Cloze Surprisal: A transformation of cloze probability, calculated as the negative logarithm of the probability. This was explored to see if a logarithmic transformation better aligned with N400 effects.
  3. Language Model (LM) Surprisal: Calculated from the probability assigned by various LMs to a target word given its preceding context. Surprisal is defined as $S(w_i) = -\log P(w_i | w_{1...i-1})$, where a higher surprisal value indicates a less predictable word.

They employed linear mixed-effects regression models to assess how well each of these measures predicted the by-trial N400 amplitudes, controlling for factors like laboratory effects and participant/item variability. They specifically compared models that included LM surprisal against those that included cloze surprisal, and also tested whether adding LM surprisal to a model already containing cloze surprisal (and vice-versa) improved its predictive power.

Datasets

The study used existing EEG data from a large-scale experiment by Nieuwland et al. [33], which included 334 participants across 9 laboratories. The dataset contained ERP data and corresponding cloze probability values for target nouns. The researchers also leveraged pre-trained language models:

  • Recurrent Neural Networks (RNNs): GRNN [34] and JRNN [35].
  • Autoregressive Transformers: Transformer-XL [39], GPT-2 [38], and GPT-3 [41].
  • Masked Language Models: BERT [36], RoBERTa [37], and ALBERT [40].

These LMs were used to generate surprisal values for the target words based on their training corpora. No new benchmarks were introduced; rather, the study aimed to compare existing measures on a well-established dataset.

Key Insights and Ideas

  • LM Surprisal Outperforms Cloze: The most striking finding is that advanced transformer-based LMs (GPT-3, RoBERTa, ALBERT) provide a significantly better fit to N400 amplitude data than traditional cloze probability. This suggests that the statistical regularities captured by these models are more aligned with how the brain processes predictability.
  • Quality Matters: Among the LMs, higher-quality models (those with more parameters and trained on larger datasets, like GPT-3, RoBERTa, and ALBERT) were better predictors of N400 amplitude. This reinforces the idea that sophisticated statistical learning is key.
  • Cloze's Limitations: The study highlights that cloze probability, while intuitive, may not fully capture the predictive processes reflected in the N400. This is partly because cloze is an aggregate measure and can be insensitive to subtle variations, especially for low-probability words.
  • Unified Explanation for N400: The findings suggest that the N400's sensitivity to seemingly disparate factors (like semantic relatedness and pure predictability) might be unified under a single mechanism: the brain's attunement to statistical patterns in language, as modeled by advanced LMs.
  • Beyond Human Intuition: The fact that LMs can predict N400 better than human-derived cloze probability, even when the stimuli were designed to be maximally favorable to cloze, implies that the brain might be more sensitive to statistical language properties than our direct introspections (as captured by cloze) suggest.

Key Results

  • Regressions using surprisal values from GPT-3, RoBERTa, and ALBERT showed a significantly better fit to the N400 data compared to regressions using cloze surprisal.
  • Adding LM surprisal (from GPT-3, BERT, RoBERTa, ALBERT) to models already including cloze surprisal significantly improved their predictive fit for N400 amplitude.
  • Conversely, adding cloze surprisal to models already including LM surprisal only improved fit for RNNs (GRNN, JRNN) and earlier transformer models (GPT-2), but not for the top-performing LMs (GPT-3, RoBERTa, ALBERT).
  • The best-performing LMs (GPT-3, RoBERTa, ALBERT) explained variance in N400 amplitude that was not captured by cloze probability.

Limitations and Future Work

The authors acknowledge that while their study provides compelling evidence, there are still questions to explore.

  • Comparison with Behavioral Data: The study focused on N400 amplitude. While cloze probability has been shown to correlate with behavioral measures like reading time, the direct comparison of LM surprisal and cloze probability for these behavioral metrics was not the primary focus. Future work could further investigate these differences.
  • LM Architectures: While they tested several LMs, the field is rapidly evolving. Future research could explore even more advanced or specialized LM architectures.
  • Stimulus Design: Although they aimed for a "level playing field," the possibility of subtle advantages for LMs in specific experimental designs remains a point for careful consideration in future studies.

The authors suggest that LM surprisal, particularly from high-quality models, should be considered a valuable tool for researchers investigating language processing, potentially replacing or supplementing traditional cloze probability measures in future ERP studies. They propose that the language comprehension system, at least as reflected by the N400, is more finely attuned to language statistics than previously thought.

Summary updated

Abstract

More predictable words are easier to process—they are read faster and elicit smaller neural signals associated with processing difficulty, most notably, the N400 component of the event-related brain potential. Thus, it has been argued that the prediction of upcoming words is a key component of language comprehension, and that studying the amplitude of the N400 is a valuable way to investigate the predictions we make. In this study, we investigate whether the linguistic predictions of computational language models (LMs) or humans better reflect the way in which natural language stimuli modulate the amplitude of the N400. One important difference in the linguistic predictions of humans versus computational LMs is that while LMs base their predictions exclusively on the preceding linguistic context, humans may rely on other factors. We find that the predictions of three top-of-the-line contemporary LMs—1) GPT-3; 2) RoBERTa; and 3) ALBERT—match the N400 more closely than human predictions. This suggests that the predictive processes underlying the N400 may be more sensitive to the statistics of language than previously thought.
BibTeX
@article{michaelov2023so,
  author = {James A. Michaelov and Seana Coulson and Benjamin K. Bergen},
  title = {{So Cloze Yet So Far: N400 Amplitude Is Better Predicted by Distributional Information Than Human Predictability Judgements}},
  journal = {IEEE Trans. Cogn. Dev. Syst.},
  year = {2023},
  url = {https://openreview.net/forum?id=4i80l37Myh},
}
Figure