Predict but Also Integrate: an Analysis of Sentence Processing Models for English and Hindi

Nina Delcaro, Luca Onnis, Raquel AlhamaACL 2024

TL;DR

This paper investigates whether human language comprehension relies solely on predicting upcoming words or also involves integrating new information with existing knowledge. By comparing a purely predictive neural network model with one that incorporates integration, the authors found that integration, particularly as measured by entropy, significantly improves the explanation of human reading behavior in both English and Hindi.

What Problem Does This Paper Address?

Human language comprehension is remarkably fast and efficient. One proposed explanation for this efficiency is prediction: the brain anticipates upcoming words or grammatical structures based on context and world knowledge. However, there's an ongoing debate about whether prediction is the only mechanism at play. An alternative or complementary mechanism is integration, where the brain efficiently processes incoming information by evaluating its probability within the current context, without necessarily predicting the exact next word. While prediction has been extensively modeled, the role and computational underpinnings of probabilistic integration in real-time language processing remain less understood. This paper aims to bridge this gap by exploring whether a model that combines prediction and integration can better explain human reading patterns than a purely predictive model.

Methods

The core of this research lies in comparing two types of Recurrent Neural Networks (RNNs) as computational models of language processing:

  • Vanilla RNN (Elman RNN): This is a standard, forward-only RNN. It's considered a model of prediction because it learns to predict the next word ($x_{t+1}$) based on the current word ($x_t$) and the network's previous state ($h_{t-1}$).
  • Delayed RNN (d-RNN): This is a variant of the RNN proposed by Turek et al. (2020). Its key innovation is a processing delay ($d$) introduced in its feedback loop. This delay means the network's weights are updated only after $d$ extra words have been processed. The authors argue that this delay allows the d-RNN to capture backward dependencies (i.e., integrate information from the past context more effectively) while still performing forward prediction. This makes it a model that can implement both prediction and integration.

To evaluate these models, the researchers extracted per-word predictor variables:

  • Surprisal: The negative log-probability of a word, reflecting its unexpectedness. This is a standard measure linked to processing difficulty.
  • Entropy: A measure of uncertainty over possible outcomes, reflecting the degree of ambiguity in the current linguistic context.

These language model-derived metrics were then used as predictors in Generalized Linear Mixed Effects Regression models (GLMER) to explain outcome variables: human reading times (specifically, first fixation duration, total fixation duration, and first-pass reading time) from eye-tracking data. They used a stepwise nested modeling approach to rigorously test whether adding the RNN or d-RNN metrics improved the model fit beyond a baseline model that included word length and position.

Datasets

The study utilized publicly accessible eye-tracking datasets for two languages:

  • English: The Multilingual Eye-tracking Corpus (MECO), containing data from 46 participants reading 112 encyclopedic sentences.
  • Hindi: The Potsdam-Allahabad Hindi Eye-tracking Corpus (PAC), with data from 30 individuals reading 83 sentences.

These datasets were chosen for their uniform, expository style, similar to Wikipedia articles. The language models (RNN and d-RNN) were trained on large corpora of Wikipedia articles for each language.

Key Insights and Ideas

  • Integration Matters: The most significant finding is that the d-RNN, which models both prediction and integration, provides a better explanation for human reading behavior than the purely predictive RNN, especially when using entropy as a predictor. This suggests that integration is a crucial component of efficient language processing.
  • Entropy as a Key Metric: The d-RNN's entropy was a significant predictor of reading times in both English and Hindi. This highlights entropy's role in capturing the uncertainty and complexity of processing, which seems to be better reflected by an integration-aware model.
  • Surprisal's Role is More Complex: While surprisal is a well-established predictor of processing difficulty, its contribution was less consistent. The RNN's surprisal improved model fit in some cases, but the d-RNN's surprisal did not offer further improvement, suggesting that the purely predictive aspect of surprisal might not fully capture the benefits of integration.
  • Language Differences: The study found different outcomes for English and Hindi, suggesting that the interplay between prediction and integration might be influenced by language typology (e.g., word order). This calls for further cross-linguistic investigation.
  • Cognitively Plausible Models: The authors specifically chose RNNs and d-RNNs over more complex architectures like transformers because they better reflect real-time, incremental processing without relying on future text, making them more cognitively plausible models for online comprehension.

Key Results

  • For English, the d-RNN's entropy significantly improved the model fit for first fixation duration and total fixation duration. The RNN's surprisal also improved model fit for all three reading time measures.
  • For Hindi, the d-RNN's entropy improved the model for total fixation duration and first-pass reading time. The RNN's entropy also improved the model. The RNN's surprisal showed an improvement for total fixation duration.
  • In general, the d-RNN's entropy was a more consistent predictor of reading times across both languages compared to the d-RNN's surprisal.

Limitations and Future Work

  • Language-Specific Differences: The differing outcomes between English and Hindi suggest that the relative importance of prediction versus integration might vary across languages. The authors propose further research into a broader spectrum of languages to understand how language structure influences this balance.
  • Model Operationalization: The specific way integration is operationalized in the d-RNN (through a delay) might be reminiscent of lookahead mechanisms in parsing. Future work could explore other computational implementations of integration.
  • Model Architectures: While RNNs and d-RNNs were chosen for their cognitive plausibility, more advanced architectures like bidirectional RNNs or transformers could potentially implement integration. However, their reliance on future text makes them less suitable for modeling real-time incremental processing. The authors suggest that exploring Long Short-Term Memory (LSTM) networks and decoder-only transformers trained unidirectionally could be fruitful avenues for future research.

Summary updated

BibTeX
@inproceedings{delcaro2024predict,
  author = {Nina Delcaro and Luca Onnis and Raquel Alhama},
  title = {{Predict but Also Integrate: an Analysis of Sentence Processing Models for English and Hindi}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure