TL;DR
This paper tackles the challenge of automatically predicting human gaze behavior (like where and how long someone looks at a word) using only linguistic features. By engineering a rich set of features related to word properties, familiarity, and context, they demonstrate that tree-based regressors, particularly CatBoost, can achieve competitive performance on this task, highlighting the strong link between language structure and visual attention.
What Problem Does This Paper Address?
Understanding human visual attention during reading, specifically eye-tracking data, offers deep insights into cognitive processes. While eye-tracking has been used in various Natural Language Processing (NLP) tasks, automatically predicting this behavior from text alone is a significant challenge. This is crucial for scaling up analyses to large datasets and for applications that could benefit from understanding reading patterns without requiring actual eye-tracking experiments. The paper addresses the need to model gaze behavior at the token (word) level within a sentence, using only linguistic information.
Methods
The core of this work lies in feature engineering. The authors extract a comprehensive set of linguistic features for each token, categorized as:
- Shallow Lexical Features: These capture basic properties of a word, such as its length, number of vowels/syllables/phonemes, whether it's a stopword, capitalized, hyphenated, or a named entity. They also consider etymological origin.
- Familiarity Features: These leverage corpus-based information to gauge how common a word is. This includes features indicating if a word is in common English word lists (like Ogden's Basic English or Google's top 10,000 words) and its frequency in various corpora (SUBTLEX, Exquisite Corpus).
- Contextual Features: These go beyond individual words to consider their surroundings. This includes Part-of-Speech (POS) tags (both Universal and Penn Treebank), dependency tags, and features derived from the sentence's dependency tree (depth, number of children).
- Language Model Features: Statistical language models (n-grams) are used to capture word co-occurrence probabilities and sentence perplexity, reflecting how "surprising" a word sequence is.
These engineered features are then fed into various machine learning regression algorithms, including Partial Least Squares Regression (PLS), Neural Networks (NN), Decision Trees (DT), Random Forests (RF), XGBoost (XG), and Light Gradient Boosting Machine (LG). Notably, they also explore Multioutput (MO) and Regressor Chain (RC) approaches to handle the prediction of multiple gaze features simultaneously.
Datasets
The primary dataset used is the ZuCo Eye-Tracking dataset. This dataset is described as containing three subsets: Train, Trial, and Test, with specific token counts provided. The paper does not introduce a new benchmark dataset but rather participates in a shared task using an existing one.
Key Insights and Ideas
- Linguistic Features are Powerful Predictors: The most striking insight is that a rich set of linguistic features can effectively predict gaze behavior, suggesting a strong, inherent link between language structure and how we visually process it.
- CatBoost Shines: The CatBoost regressor, with its ability to handle categorical features directly and its permutation-driven boosting, emerged as the top performer among the tested algorithms. This highlights the advantage of algorithms that can leverage the nature of linguistic features.
- Feature Importance: The paper's analysis of feature elimination reveals that all engineered features play a role, with
Relpos(relative position of the token in the sentence) being identified as particularly important. This underscores the significance of both word-level properties and their sequential placement. - Context Matters, But Capturing Variation is Hard: While contextual features are included, the authors note that predicting gaze behavior for the same word can vary significantly across different contexts. Their current model doesn't fully capture this nuanced variation.
Key Results
The paper reports performance using Mean Absolute Error (MAE). The CatBoost+Multioutput (CB+MO) model achieved fourth position in terms of MAE on the ZuCo Dataset. While not the absolute top performer, it was within 0.14 MAE of the best. The paper also shows that removing individual features generally increased the MAE, indicating the collective importance of the engineered feature set. Table 4 provides specific examples of system predictions versus gold labels for certain tokens, illustrating both successes and areas where predictions deviate.
Limitations and Future Work
The authors acknowledge that their current system does not fully capture the variation in Eye-Tracking features for the same words appearing in different contexts. This is identified as a key limitation. Future work will focus on incorporating more features to better represent the context of the target token, aiming to improve the model's ability to handle this contextual variability.
Summary updated