An Information-Theoretic Analysis of Targeted Regressions during Reading

Ethan Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell

TL;DR

This paper introduces an information-theoretic framework to explain why readers make regressive eye movements (go backward) during reading. They found that regressions are more likely to occur between words that are statistically associated (high pointwise mutual information), supporting the "reactivation hypothesis" and suggesting readers revisit words to confirm their understanding.

What Problem Does This Paper Address?

Reading involves not just moving forward but also occasional backward eye movements, known as regressions. While these regressions are common (5-20% of saccades), their underlying causes are not well understood. Two main hypotheses exist: "reactivation" (readers revisit words to confirm their understanding of associated words) and "reanalysis" (readers revisit words to resolve confusion or ambiguity). The challenge has been to quantitatively test these qualitative hypotheses and understand their generalizability across languages and reading contexts.

Methods

The authors operationalize the reactivation and reanalysis hypotheses using pointwise mutual information (PMI). PMI measures how much more likely two words are to appear together given their context.

  • Reactivation Hypothesis: Predicts regressions between words with high positive PMI (associated words).
  • Reanalysis Hypothesis: Predicts regressions between words with low negative PMI (disassociated words).

They also introduce an expectation-based measure (E[PMI]) to account for uncertainty in language processing, where the expectation is taken over possible word identities. They use large, pre-trained language models (like BERT) to estimate these PMI values from text. These PMI-based statistics, along with other linguistic features (like word frequency, length, and surprisal), are then used as predictors in zero-inflated Poisson (ZIP) models to predict the number of regressions between word pairs in sentences.

Datasets

The study uses several datasets:

  • Study One (English): Three corpora of naturalistic reading data:
  • Provo Corpus
    • UCL Corpus
    • Dundee Corpus
  • Study Two (Multilingual): A subset of the Multilingual Eye Movement Corpus (MECO) including data from six languages across three families:
  • Indo-European: German, Spanish, Italian
    • Uralic: Finnish
    • Turkic: Turkish
    • (Russian was also included in the MECO subset)

Key Insights and Ideas

  • Regressions as Confirmation: The strongest finding is that regressions are consistently predicted by positive PMI, supporting the idea that readers revisit words to confirm associations and build confidence in their understanding.
  • Reactivation Over Reanalysis: The evidence strongly favors the reactivation hypothesis, with negative PMI (associated with reanalysis) showing little predictive power.
  • Expectations Matter: The expected value of PMI (E[PPMI]) was often a better predictor than raw PMI, suggesting readers' decisions to regress are influenced by their expectations about word associations, especially under uncertainty.
  • Crosslinguistic Consistency: The findings hold across multiple languages and corpora, suggesting that the information-theoretic principles driving regressions are universal.
  • PMI as a General Predictor: This work demonstrates that PMI, a measure of word association, can be a powerful, general predictor of a language processing behavior (regressions), extending the reach of information-theoretic principles.

Key Results

  • PMI Predicts Regressions: Positive PMI coefficients in the ZIP models consistently predicted an increase in regressions. This effect was observed across all English corpora and in all tested languages.
  • PPMI and E[PPMI] are Key: Positive PMI (PPMI) and its expected value (E[PPMI]) significantly improved model predictions (higher Delta Log-Likelihood) compared to baseline models. E[PPMI] generally showed stronger predictive power than PPMI.
  • NPMI and E[NPMI] Lack Predictive Power: Negative PMI (NPMI) and its expected value (E[NPMI]) did not consistently predict regressions, providing little support for the reanalysis hypothesis.
  • Baseline Predictors: Consistent with prior work, word surprisal (difficulty) and source word frequency (infrequent words) were significant predictors of regressions. Distance between words also played a role, with regressions being less likely between distant words.

Limitations and Future Work

  • Controlled Experiments: The study primarily used corpus data. Future work could involve controlled experimental manipulations to further validate causal relationships.
  • Subtle Language Differences: While crosslinguistic consistency was high, some variations in the strength of PMI's predictive power across languages were observed (e.g., stronger effects in German and Finnish). Further investigation is needed to understand these differences.
  • Uncertainty Strategies: The paper explored two strategies (certainty with PPMI, uncertainty with E[PPMI]). Future work could investigate more nuanced uncertainty representations, such as annealed probability distributions over candidate words.
  • Wrap-up Effects: The study noted potential wrap-up effects (regressions at sentence ends) might influence distance effects, suggesting further analysis excluding these cases.

Summary updated

Abstract

Regressions, or backward saccades, are common during reading, accounting for between 5% and 20% of all saccades. And yet, relatively little is known about what causes them. We provide an information-theoretic operationalization for two previous qualitative hypotheses about regressions, which we dub reactivation and reanalysis. We argue that these hypotheses make different predictions about the pointwise mutual information or pmi between a regression’s source and target. Intuitively, the pmi between two words measures how much more (or less) likely one word is to be present given the other. On one hand, the reactivation hypothesis predicts that regressions occur between words that are associated, implying high positive values of pmi. On the other hand, the reanalysis hypothesis predicts that regressions should occur between words that are disassociated with each other, implying negative, low values of pmi. As a second theoretical contribution, we expand on previous theories by considering not only pmi but also expected values of pmi, E[pmi], where the expectation is taken over all possible realizations of the regression’s target. The rationale for this is that language processing involves making inferences under uncertainty, and readers may be uncertain about what they have read, especially if a previous word was skipped. To test both theories, we use contemporary language models to estimate pmi-based statistics over word pairs in three corpora of eye tracking data in English, as well as in six languages across three language families (Indo-European, Uralic, and Turkic). Our results are consistent across languages and models tested: Positive values of pmi and E[pmi] consistently help to predict the patterns of regressions during reading, whereas negative values of pmi and E[pmi] do not. Our information-theoretic interpretation increases the predictive scope of both theories and our studies present the first systematic crosslinguistic analysis of regressions in the literature. Our results support the reactivation hypothesis and, more broadly, they expand the number of language processing behaviors that can be linked to information-theoretic principles.
BibTeX
@inproceedings{wilcox2024information,
  author = {Ethan Wilcox and Tiago Pimentel and Clara Meister and Ryan Cotterell},
  title = {{An Information-Theoretic Analysis of Targeted Regressions during Reading}},
  booktitle = {Cognition},
  year = {2024},
  doi = {10.1016/j.cognition.2024.105765},
}
Figure