TL;DR
This paper explores predicting human reading patterns using both traditional oculomotor control models and modern machine learning. The authors found that combining these approaches yields the most accurate predictions for eye-tracking variables, suggesting that a hybrid approach offers a more comprehensive understanding of reading behavior than either method alone.
What Problem Does This Paper Address?
Understanding how humans read is a long-standing challenge in cognitive science and linguistics. Eye-tracking research has revealed that various factors influence where and for how long we fixate our gaze on a word. These factors include word characteristics like frequency (how often a word appears in general usage) and length, as well as its predictability within a sentence (how likely it is to appear given the preceding words). While traditional computational models like the EZ Reader have been developed to simulate these oculomotor control mechanisms, they often struggle to incorporate higher-level linguistic information like syntax and semantics. Conversely, machine learning models can achieve high predictive power but often lack interpretability. This paper addresses the gap by investigating how well traditional models, machine learning models, and their combinations can predict specific eye-tracking features, aiming to leverage the strengths of both approaches.
Methods
The researchers employed three main modeling strategies:
- Traditional Oculomotor Control Model (EZ Reader): This rule-based model simulates eye movement control during reading. It takes word length, frequency, and sentential predictability as inputs and outputs various eye-tracking metrics.
- Linear Baseline Model: A standard least-squares regression model with L2 regularization. This model incorporated a wide range of features, including word-level characteristics (frequency, length, predictability), linguistic features (POS tags, dependency relations), named entity tags, and punctuation indicators. Crucially, it also included outputs from the EZ Reader model as input features.
- LSTM Model: A recurrent neural network model designed to capture sequential dependencies. This model also utilized word length, predictability, frequency, and sentence position. Similar to the linear model, it incorporated outputs from the EZ Reader model.
The study also explored different combinations of these features to understand their individual and collective impact on prediction accuracy.
Datasets
The primary dataset used for training and evaluation was the ZuCo Corpus (ZuCo 1.0 and ZuCo 2.0), a collection of simultaneous EEG and eye-tracking data from participants reading natural sentences. The paper also mentions using a large news corpus (approximately 2.7 million articles) for computing word frequencies and the Provo Corpus, which provides human predictability scores derived from cloze tasks.
Key Insights and Ideas
- Hybrid Approach Wins: Combining traditional oculomotor models (like EZ Reader) with machine learning models (linear regression and LSTM) consistently improved prediction accuracy for eye-tracking variables. This suggests that neither approach is sufficient on its own.
- Feature Importance: The study reinforces the well-established finding that word frequency, length, and sentential predictability are crucial predictors of reading behavior.
- EZ Reader's Contribution: Surprisingly, the EZ Reader's output features, particularly its single fixation duration prediction (EZ-SFD), proved to be valuable inputs for the machine learning models, contributing to improved accuracy.
- Interpretability vs. Performance Trade-off: While machine learning models generally outperformed traditional models, the paper acknowledges the "holistic approach" of ML models can make their underlying mechanisms less explainable.
Key Results
The paper reports official Mean Absolute Error (MAE) scores for their models in the CMCL 2021 shared task.
- The Linear Baseline model achieved the 10th best score in the competition.
- The LSTM model performed slightly worse than the linear model but provided more insights into feature effects.
- When comparing feature combinations, models that included a combination of frequency, predictability, word length, and EZ-SFD achieved the best MAE values for four out of five dependent variables. For instance, the model incorporating these features showed significantly better performance for predicting the number of fixations (nFix) compared to models using features in isolation.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Operational Assumptions: The study relied on certain operational assumptions, particularly in handling "Infinity" values from the EZ Reader model. Future research should refine statistical analysis to compare model outputs more rigorously.
- Feature Interaction: While the study explored feature combinations, a deeper investigation into the interaction of features, especially the influence of word location within a sentence, is needed.
- Model Design and Reading Research: Future work should focus on developing ML models that more closely align with established findings in reading research, such as the Gaze Contingent Moving Window and boundary paradigms.
- Explainability: Further research could explore methods to improve the explainability of machine learning models used for oculomotor control.
Summary updated