TL;DR
This paper provides large-scale evidence that the difficulty of reading a word is logarithmically related to its predictability in context. This finding supports the "cost of probabilistic inference" view of language comprehension, suggesting our brains are constantly updating interpretations of sentences, and this process has a computational cost that increases with uncertainty.
What Problem Does This Paper Address?
When we read, our brains don't just passively process words; they actively predict what's coming next. This prediction process influences how quickly and easily we comprehend text. However, there's a debate about how this predictability affects processing. One camp, the "FACILITATION" view, suggests that predictable words are processed more easily, leading to a linear relationship between predictability and processing ease. The other camp, the "COST" view, argues that predictability effects arise from the computational effort of updating our understanding of the sentence based on new information. This view predicts a logarithmic or even superlogarithmic relationship between predictability and processing difficulty. Previous studies have yielded mixed results, making it unclear which theory better explains human reading. This paper aims to resolve this by using a large-scale, data-driven approach.
Methods
The authors employed a novel statistical modeling technique called Continuous-Time Deconvolutional Regression Networks (CDRNNs). This method is powerful because it can model complex, nonlinear relationships between predictors (like word predictability) and response variables (reading times) over continuous time, without making restrictive assumptions about linearity or additivity. CDRNNs learn an "impulse response function" that describes how a word's predictability affects reading time at different delays.
To estimate word predictability, the researchers used a variety of statistical language models, including traditional n-gram and probabilistic context-free grammar (PCFG) models, as well as modern transformer-based models like GPT-2, GPT-J, and GPT-3. They then compared how well these different predictability measures, when modeled with CDRNNs, predicted actual reading times from large datasets.
Datasets
The study utilized six publicly available, large-scale naturalistic reading datasets:
- Brown self-paced reading (SPR) dataset: Participants read text at their own pace, pressing a button to reveal the next word.
- Dundee eye-tracking (ET) dataset: Recorded participants' eye movements as they read.
- GECO eye-tracking (ET) dataset: Similar to Dundee, but with a focus on bilingual readers.
- Natural Stories self-paced reading (SPR) dataset: A collection of narrative and nonfiction texts designed to include rare syntactic constructions.
- Natural Stories Maze dataset: Participants made forced choices between the correct next word and a distractor.
- Provo eye-tracking (ET) dataset: Another eye-tracking corpus.
These datasets represent different reading modalities (self-paced reading and eye-tracking) and vary in size and content, allowing for robust generalization.
Key Insights and Ideas
- Logarithmic is the Winner: The most striking finding is the consistent support for a logarithmic relationship between word predictability and processing difficulty across multiple datasets and language models. This strongly favors the "cost of probabilistic inference" theory.
- Surprisal is Key: The paper emphasizes that surprisal (the negative log probability of a word) is a more effective predictor of reading difficulty than raw probability. This is because small changes in probability at low probabilities (high uncertainty) lead to large changes in surprisal, which the logarithmic model captures.
- GPT-2 Shines: Surprisingly, the smaller GPT-2 language model often outperformed larger models like GPT-3 in predicting human reading times. This suggests that simply having more parameters or better perplexity doesn't automatically translate to better psycholinguistic modeling, hinting at an "inverse scaling" phenomenon.
- Cloze Norms vs. Language Models: While cloze tasks (where humans predict words) are often considered the gold standard for predictability, the study found that statistical language models (especially GPT-2) can be equally or even more effective at predicting reading times, particularly for low-probability words where cloze estimates are less reliable.
Key Results
- Statistical models constrained to a logarithmic effect of word predictability (linear effect of surprisal) significantly outperformed models assuming a linear effect (FACILITATION view) across all datasets.
- Models assuming a superlogarithmic effect (consistent with the Uniform Information Density hypothesis) did not consistently outperform the logarithmic model, and in some cases, performed worse.
- The unconstrained CDRNN models (which could learn any functional form) also converged on estimates consistent with a logarithmic effect, reinforcing the finding.
- GPT-2 surprisal estimates showed superior predictive power compared to n-gram, PCFG, GPT-J, and GPT-3 models in most comparisons.
Limitations and Future Work
- Preview Validity: The authors acknowledge that findings from eye-tracking studies involving "invalid parafoveal preview" (where the previewed word is incorrect) can sometimes contradict their results. They suggest this might be due to strategic adaptations in processing or differences in how early vs. late processing measures are affected. Future work could explore these interactions more deeply.
- Dataset-Specific Effects: While the overall trend is logarithmic, some specific datasets (like Natural Stories SPR) and larger language models (GPT-J, GPT-3) showed tendencies towards superlogarithmic effects. The authors caution against overinterpreting these, suggesting they might be dataset-specific or artifacts of model training.
- Model Selection: The finding that GPT-2 outperforms larger models raises questions about how to best select language models for psycholinguistic research. Future work could explore the "inverse scaling" phenomenon further and identify optimal model characteristics beyond simple perplexity.
- Other Predictors: The study focused on word predictability. Future research could integrate other known predictors of reading difficulty (e.g., word frequency, repetition priming) within the CDRNN framework to build more comprehensive models of language comprehension.
Summary updated