TL;DR
This paper bridges neurolinguistics and NLP by comparing human word predictability (cloze-task) with computational models (N-grams, LSA, Word2Vec, FastText) to understand reading behavior. They found that N-gram models better predict fixated words, while word embeddings better capture the predictability of upcoming words, offering insights into how NLP can better model human reading.
What Problem Does This Paper Address?
Reading is a complex cognitive process where we constantly predict upcoming words to guide our eye movements and integrate information. "Predictability" of a word—how likely it is to appear given the preceding context—is a key factor influencing reading speed and comprehension. Traditionally, human predictability is measured via expensive "cloze-tasks," where participants fill in missing words. This is a bottleneck for large-scale studies and for integrating computational models into reading research. The paper addresses the gap of computationally estimating word predictability in a way that aligns with human cognitive processes during reading, particularly in longer texts, and how these computational estimates can explain human eye movement data.
Methods
The researchers employed a multi-pronged approach:
- Human Predictability Estimation: They conducted a large-scale online cloze-task experiment with thousands of participants to estimate word predictability for a corpus of Spanish short stories. This created a benchmark for human predictability.
- Computational Predictability Models: They implemented and tested four NLP algorithms:
- N-grams: Specifically, a 4-gram model with a "cache" mechanism (incorporating recently read words) was used. Parameters for smoothing and interpolation were optimized.
- Latent Semantic Analysis (LSA): Used to create 300-dimensional word embeddings from a large corpus.
- Word2Vec: Also used to generate 300-dimensional embeddings.
-
FastText: Similar to Word2Vec but incorporates character n-grams for better handling of out-of-vocabulary words. For LSA, Word2Vec, and FastText, they explored various context window sizes to estimate predictability as the cosine similarity between the target word and its preceding context.
-
Eye-Tracking Data Analysis: They recorded eye movements (specifically gaze duration) of participants reading the same Spanish stories.
- Statistical Modeling: Linear Mixed Models (LMMs) were used to analyze the relationship between word predictability (both human and computational) and gaze duration, controlling for other linguistic variables like word frequency and length. They compared different model configurations using AIC and ANOVA.
Datasets
- Human Predictability Dataset: A new dataset of cloze-task responses for eight Spanish short stories (totaling over 25,000 words) was created. This dataset is made available for future research.
- NLP Training Corpus: A large corpus of 2082 Spanish books (over 100 million words) was used to train the N-gram, LSA, Word2Vec, and FastText models. This corpus did not include the stories used in the cloze-task or eye-tracking experiments.
- Eye-Tracking Dataset: Eye movement data from 36 participants reading the eight Spanish short stories was collected and analyzed.
Key Insights and Ideas
- N-grams for Fixated Words, Embeddings for Upcoming Words: A key finding is the differential performance of computational models. N-gram models, particularly with a cache, were effective at mimicking the predictability of the current (fixated) word. In contrast, word embeddings (LSA, FastText) were better at capturing the predictability of the next word.
- Long Text Predictability Differs from Sentence-Level: The negative effect of next-word predictability on gaze duration observed in this study for long texts was not consistently found in previous sentence-reading studies. This suggests that predictability in longer texts might involve different cognitive processes, potentially influenced by higher-level semantic integration.
- Bridging Cognitive Science and NLP: The study demonstrates a valuable approach to using human cognitive data (eye movements) to evaluate and refine NLP models, and conversely, using NLP models to gain insights into human cognition. This interdisciplinary approach can lead to more human-like AI and a deeper understanding of reading.
- N-gram+Cache as a Strong Predictor: The N-gram+cache model showed a significant contribution to explaining gaze duration, even outperforming human cloze-predictability in some aspects when considered as a replacement. However, it also reduced the significance of word frequency, suggesting a strong correlation between these two factors.
Key Results
- Human Predictability: Cloze-predictability correlated positively with repetition number, log frequency, and inversely with word length, as expected.
- N-gram+Cache Model: This model significantly predicted gaze duration and was found to partially capture aspects of human cloze-predictability for the fixated word.
- Word Embeddings (LSA, FastText): These models were more successful at mimicking the predictability of the following word (N+1) than the fixated word (N). FastText, in particular, showed promise for predicting the next word.
- Next-Word Predictability: Cloze-predictability of the next word (N+1) had a significant, and surprisingly negative, effect on the gaze duration of the current word (N). This effect was better explained by word embedding models.
Limitations and Future Work
- Cloze-Task Limitations: The authors acknowledge that their cloze-task, where participants answered every 30 words, provided more context than typical sentence-level tasks. This might have influenced the captured predictability.
- Computational Cost: While FastText and Word2Vec are powerful, training them on very large corpora can be computationally intensive.
- Future Directions: The authors suggest extending this approach to more advanced NLP models like Transformers (BERT, ElMo, ULM-FiT), which are computationally demanding but could offer even richer representations. They also propose fine-tuning these models using implicit human responses (like eye movements) to optimize predictions and better align them with human reading processes. Further research could also explore how these computational predictions influence other cognitive measures like pupil dilation.
Summary updated