TL;DR
This paper presents a winning system for the CMCL 2021 Shared Task on Eye-Tracking Data Prediction. The authors developed a Gradient Boosting Decision Tree (LightGBM) model that leverages a rich set of linguistic and psycholinguistic features to predict human gaze patterns during reading, outperforming deep learning models and achieving top ranks in the challenge.
What Problem Does This Paper Address?
Understanding how humans process written language is a fundamental goal in cognitive science and natural language processing. Eye-tracking data, which records where and for how long people look at words while reading, provides a rich window into this process. Predicting these gaze patterns can help us build better models of language comprehension, improve reading aids, and even understand cognitive impairments. The challenge lies in accurately modeling the complex interplay of factors that influence where a reader's eyes land, from word properties to sentence context. This paper tackles the problem of predicting specific eye-tracking metrics (like fixation counts and durations) for individual words in a sentence, using machine learning.
Methods
The core of the system is a LightGBM (Light Gradient Boosting Machine) model. LightGBM is a gradient boosting framework that uses decision trees and is known for its efficiency and ability to handle large datasets. The authors chose LightGBM because it doesn't require extensive feature preprocessing (like logarithmic transformations) and can directly optimize for the Mean Absolute Error (MAE) metric, which was used for evaluation in the shared task.
The system was trained using five separate LightGBM models, one for each of the five eye-tracking metrics to be predicted. The key innovation lies in the extensive feature engineering. The models were fed a diverse set of features, including:
- Word-level features: Length of the target word, preceding, and following words.
- Positional features: The word's rank in the sentence and its relative position.
- Linguistic features: Part-of-speech (POS) tags and lemmas for the target and preceding words.
- Corpus frequency features: Various measures of word frequency from multiple large corpora (e.g., BNC, Google ngrams).
- Lexical norm features: Age of Acquisition and Familiarity ratings from established psycholinguistic databases.
- Behavioral measures: Indices from the English Lexicon Project (ELP), including reaction times and accuracies from lexical decision and naming tasks.
- Bigram association measures: These "collgrams" capture the association strength between a word and its preceding word, going beyond simple co-occurrence frequencies. This is a novel application of these measures to eye-tracking prediction.
The models were optimized using a 5-fold cross-validation procedure, and early stopping was employed to prevent overfitting.
Datasets
The eye-tracking data for this shared task was extracted from the Zurich Cognitive Language Processing Corpus (ZuCo 1.0 and ZuCo 2.0). This corpus contains gaze data for 991 sentences read by 18 participants. The dataset was split into a training set of 800 sentences and a test set of 191 sentences.
The features were derived from various external resources, including:
- British National Corpus (BNC)
- Facebook and Twitter frequency norms
- USENET Orthographic Frequencies
- Hyperspace Analogue to Language (HAL) frequency norms
- Google's ngram corpora
- Glasgow Norms (Age of Acquisition, Familiarity)
- English Lexicon Project (ELP)
Key Insights and Ideas
- Feature Engineering is King: The success of the system hinges on the rich and diverse set of linguistic and psycholinguistic features. The paper demonstrates that combining these different types of information is crucial for accurate gaze prediction.
- Bigram Association Measures Shine: The inclusion of "collgrams" (bigram association measures) was a novel approach for this task and proved to be highly useful, suggesting that the contextual relationship between words is a strong predictor of gaze behavior.
- LightGBM's Efficiency: The choice of LightGBM proved effective, allowing for efficient training and optimization, and its insensitivity to monotonic transformations simplified feature preprocessing.
- Feature Redundancy and Importance: Ablation studies revealed that while some features (like POS tags and lemmas) were not directly useful on their own, others like word length and position were essential. The paper hints at potential redundancy among feature sets, suggesting future work in feature selection.
- Optimizing for MAE: The system's success was partly due to optimizing for MAE, especially for the
fixPropmetric, which had a much larger mean and variance compared to other metrics.
Key Results
The authors' system, LAST, achieved first place in the CMCL 2021 Shared Task on Eye-Tracking Data Prediction. Specifically, their best run (Run 2) achieved an average MAE of 3.8134. They also ranked second and third with their other submissions. The system outperformed all deep-learning based systems participating in the challenge.
The ablation studies showed that:
- Removing behavioral data, ELP characteristics, frequencies, or bigram AMs led to a performance drop of around 1.7-4.35%.
- Removing length features caused a significant drop of 4.35%, while removing position features led to the largest drop of 7.39%.
- Optimizing LightGBM parameters specifically for MAE (compared to RMSE) was beneficial, especially for the GPT metric.
Limitations and Future Work
- Feature Redundancy: The authors acknowledge that there might be significant redundancy between different feature sets (e.g., frequency, behavioral data, bigram AMs). Future work could involve exploring feature selection to identify the most impactful and non-redundant features.
- Psycholinguistic Interpretation: While the features are derived from psycholinguistic principles, a deeper analysis to directly relate the findings to existing literature on eye fixation factors is suggested but limited by space.
- Parameter Optimization: While the parameter optimization was beneficial, the paper notes that a specific model for each DV might have yielded slightly better results, though this was not explored in depth.
- Comparison to Deep Learning: Although the LightGBM model outperformed deep learning models in this specific task, the paper doesn't delve into why this might be the case, leaving room for further investigation into the strengths of tree-based models for this type of structured data.
Summary updated