On the Role of Context in Reading Time Prediction

Andreas Opedal, Eleanor Chodroff, Ryan Cotterell, Ethan WilcoxEMNLP 2024 · DOI

Abstract

Surprisal theory—the idea that reading time is driven by how "surprising" a word is in context—is a cornerstone of modern psycholinguistics. This paper argues that the role of context has been historically overestimated because surprisal is heavily confounded with simple word frequency. By orthogonalizing these variables to isolate the unique contribution of context across 11 languages, the authors reveal that context explains far less variance in reading times than previously thought, while frequency and word length remain the primary drivers of processing effort.

Introduction

In psycholinguistics, surprisal theory posits that the cognitive effort required to process a word is an affine function of its information content (surprisal) given the preceding text. While many studies find that surprisal predicts reading times well, there is a significant confound: surprisal is highly correlated with unigram frequency (how common a word is in general, regardless of context).

Because frequency is already known to be a massive driver of reading speed, this research asks a critical question: How much of the "surprisal effect" is just frequency in disguise? If we strip away everything that frequency already explains, what is left for "context" to do?

Methodology

The authors introduce a rigorous framework to disentangle these effects using linear algebra and information theory:

  1. The Collinearity Identity: They prove that contextual surprisal, Pointwise Mutual Information (PMI), and unigram frequency are linearly dependent ($PMI = Frequency - Surprisal$). This means any linear model using two of these variables is mathematically equivalent to a model using the other two.
  2. Orthogonalized Surprisal: To isolate the "pure" effect of context, the authors project the surprisal vector onto the orthogonal complement of the frequency vector. This creates a new predictor that is, by construction, uncorrelated with frequency.
  3. LMG Analysis: Instead of relying solely on regression coefficients, they use the Lindeman, Merenda, and Gold (LMG) technique. This method averages the proportion of variance explained by a predictor across all possible orderings of predictors in the model, providing a cleaner look at relative importance.

Datasets and Experimental Setup

The study utilizes the Multilingual Eye-movement Corpus (MECO), which contains word-level eye-tracking measurements (gaze duration) from readers across 11 diverse languages, including English, Russian, Hebrew, and Korean. Probability estimates (surprisal) were generated using mGPT (a multilingual GPT-2), while baseline unigram frequency estimates were sourced from the Wordfreq library.

Key Results and Analysis

The primary finding is a significant reduction in the perceived importance of context once frequency is accounted for. Across all 11 languages, the proportion of variance explained by "pure" context (orthogonalized surprisal) is much smaller than that of standard surprisal.

Relative importance (LMG) of predictors across 11 languages for Surprisal, PMI, and Orthogonalized models.

As shown in the figure above, in most languages, orthogonalization shifts context from being a primary predictor to being less important than the frequency of the previous word (spillover effects). Frequency and word length consistently remain the "heavyweights" of reading time prediction.

The Model Size Paradox

This analysis offers an explanation for a recent paradox: larger Language Models (LLMs) often predict human reading times worse than medium-sized ones. Large models are exceptionally good at context, effectively decoupling it from frequency. If humans rely heavily on the frequency-correlated parts of context to read quickly, a "perfect" contextual model will naturally diverge from human behavior.

Discussion and Limitations

While the study provides a robust challenge to the current weight given to surprisal, it acknowledges certain limitations. The analysis used fixed-effects linear regression on averaged data; future work should replicate these findings using mixed-effects models to account for individual reader variation.

Furthermore, the persistent dominance of word length suggests that the physical constraints of planning and executing eye movements (saccades) might be a more significant bottleneck in reading than the cognitive information processing described by surprisal theory. Finally, while the study covers 11 languages, the unique results in the Korean data suggest that script and writing systems may further influence how context and frequency interact.

Summary updated

BibTeX
@inproceedings{opedal2024role,
  author = {Andreas Opedal and Eleanor Chodroff and Ryan Cotterell and Ethan Wilcox},
  title = {{On the Role of Context in Reading Time Prediction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.179},
}
Figure