A large-scale study of the effects of word frequency and predictability in naturalistic reading

Cory ShainNAACL 2019 · DOI

Introduction

In psycholinguistics, a central debate concerns how the human brain processes words during reading. Two primary factors are known to influence gaze duration: Frequency (how often a word appears in a language) and Predictability (how likely a word is given its specific context). Traditional theories often treat these as distinct cognitive modules: a memory-based lexical retrieval system for frequency and a context-driven predictive coding system for predictability.

While artificial, lab-constructed sentences often show independent effects for both, these stimuli lack the ecological validity of real-world reading. Furthermore, because common words are frequently predictable, untangling their effects in naturalistic text is statistically challenging. This study investigates whether these effects remain truly separable when analyzing large-scale, natural reading data.

Methodology

To address the complexities of naturalistic reading, the study utilizes Deconvolutional Time Series Regression (DTSR). Unlike standard models that assume instantaneous effects, DTSR models "temporal diffusion"—the reality that the processing difficulty of one word often spills over into subsequent fixations. By estimating an impulse response function for each predictor, the model captures how linguistic features influence reading time over several seconds.

Estimated impulse response functions for various predictors using DTSR, illustrating the temporal diffusion of processing effects.

Predictability was estimated using a 5-gram language model (KenLM) trained on the Gigaword 3 corpus, measured as surprisal. The study analyzed three major corpora—Natural Stories, Dundee, and UCL—comprising over one million fixations from 243 subjects. To ensure robustness, the author employed ablative testing on held-out data, using permutation tests to determine which features significantly improved the model's generalizability.

Key Results: The Dominance of Predictability

The study's most striking finding is the potential collapse of the frequency effect when predictability is robustly modeled. While both 5-gram surprisal (predictability) and unigram log-probability (frequency) were significant predictors when tested in isolation, they performed very differently in combination.

In the "Over and Above" test, adding predictability to a model that already included frequency resulted in significant improvements across all datasets ($p < 0.0001$). However, the inverse was not true: when frequency was added to a model that already accounted for predictability, it failed to significantly improve the model's performance in the pooled data ($p = 0.1515$).

Model comparison p-values demonstrating that predictability explains the variance previously attributed to frequency, while frequency fails to explain variance beyond predictability.

Discussion and Implications

These results support the "Epiphenomenon" Hypothesis, suggesting that word frequency effects may not be a primary driver of processing difficulty. Instead, they may be a byproduct of a brain fundamentally optimized for prediction. The discrepancy between these findings and previous artificial experiments may stem from the lack of discourse context in constructed stimuli, which might force the brain to engage domain-general executive control rather than specialized linguistic prediction mechanisms.

While the author cautions that "absence of evidence is not evidence of absence"—frequency effects might simply be too small to detect alongside the massive impact of predictability—the study suggests that much of what was previously attributed to lexical memory may actually be a function of predictive processing. Future work using modern neural language models may further clarify the extent to which prediction subsumes traditional lexical factors.

Summary updated

BibTeX
@inproceedings{shain2019large,
  author = {Cory Shain},
  title = {{A large-scale study of the effects of word frequency and predictability in naturalistic reading}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1413},
}
Figure