TL;DR
This paper presents a system for predicting human eye movements during reading by combining powerful language model embeddings (BERT) with traditional linguistic and behavioral features. The research demonstrates that a hybrid approach significantly outperforms models relying solely on either BERT or traditional features, highlighting the importance of a multi-faceted understanding of text for predicting reading behavior.
What Problem Does This Paper Address?
Understanding how humans read is a fundamental challenge in cognitive science and natural language processing. Eye-tracking studies provide rich data on reading behavior, but predicting these movements accurately based on text alone is complex. Traditional methods often rely on handcrafted linguistic features, while newer approaches leverage deep learning models like BERT. This paper tackles the question of how to best combine these two paradigms to achieve more accurate predictions of eye-tracking measures, such as fixation durations and total reading times. The goal is to build models that can better explain and predict the cognitive processes involved in reading.
Methods
The authors employ a three-pronged experimental approach to predict five key eye-tracking measures: number of fixations (nFix), first fixation duration (FFD), go-past time (GPT), total reading time (TRT), and fixation proportion (fixProp).
- Surface, Linguistic, and Behavioral (SLB) Features: They extract a comprehensive set of features, including:
- Surface Features: Word and sentence length, word length-sentence length ratio.
- Linguistic Features: Part-of-speech (POS) tags, word type, named entity recognition, dependency relations, semantic similarity measures (word to sentence, word to previous word, word to preceding words), and semantic surprisal (derived from a bigram language model).
- Behavioral Features: Age of acquisition, prevalence, valence, arousal, dominance, and concreteness.
- BERT Embeddings: They utilize two types of BERT embeddings:
- Pre-trained BERT: Embeddings from the base BERT model, specifically from layer 11, which is found to capture semantic properties well.
- Fine-tuned BERT: BERT models are fine-tuned separately for each of the five eye-tracking measures. Embeddings from these fine-tuned models are then used as features.
- Model Architectures: Three regression models are used: Linear Regression (LR), Decision Tree Regressor (DT), and Random Forest Regressor (RF). The Random Forest Regressor generally yielded the best performance.
The experiments involve training models using only SLB features, only BERT embeddings, and finally, a combination of fine-tuned BERT embeddings with a carefully selected subset of SLB features.
Datasets
The research utilizes the ZuCo 1.0 and ZuCo 2.0 corpora, which contain eye-tracking data collected during reading tasks on English materials. These corpora are the basis for the CMCL 2021 Shared Task on eye-tracking prediction.
Key Insights and Ideas
- Hybrid is Better: Combining BERT embeddings with carefully selected SLB features significantly outperforms models that rely solely on either BERT or SLB features. This suggests that a holistic approach, integrating deep semantic understanding with explicit linguistic and cognitive cues, is crucial for accurate eye-movement prediction.
- SLB Features Still Reign Supreme (in some ways): Surprisingly, the paper finds that training solely on SLB features can yield better results than training solely on pre-trained or fine-tuned BERT embeddings. This highlights the continued relevance and predictive power of well-engineered traditional features.
- Word Length is King: Word length emerges as the most consistently strong predictor across all five eye-tracking measures. This reinforces existing theories that longer words require more processing time.
- Expectedness Matters: Features related to word expectedness, such as frequency score and semantic similarity to preceding words, are also highly predictive. This aligns with the idea that readers anticipate upcoming words, and deviations from this expectation influence reading time.
- Surprisal and GPT Link: Semantic surprisal shows a particularly strong impact on predicting Go-Past Time (GPT). This suggests that encountering unexpected words can trigger re-reading or backward eye movements to re-process the preceding context.
- Syntactic Features Less Impactful: Standard syntactic features like POS tags and dependency relations, while informative for many NLP tasks, did not show a strong predictive effect on eye movements in this study, with the exception of numeral tokens in specific contexts.
Key Results
The best performing model, which combined fine-tuned BERT embeddings with a reduced set of SLB features, achieved a Mean Absolute Error (MAE) of 3.982 on the development set for the nFix measure. Across all five measures, the best hybrid model achieved an average MAE of 4.059 on the development set. The paper reports that their system ranked 8th out of 13 models submitted to the CMCL 2021 Shared Task.
Limitations and Future Work
- Limited Scope of SLB Features: While the SLB feature set is rich, the authors acknowledge that it doesn't encompass all possible linguistic or cognitive factors.
- Data Specificity: The strong performance of numeral tokens in predicting GPT and TRT is attributed to the biographical nature of the ZuCo dataset. This suggests that feature importance might be domain-specific.
- Future Work: The authors suggest further exploration into more sophisticated ways of combining BERT embeddings with SLB features. They also implicitly point towards the need for more diverse datasets to generalize findings across different text genres and reading contexts. The success of fine-tuning BERT for specific prediction tasks also opens avenues for more targeted model development.
Summary updated