TL;DR
This paper presents a novel approach to predicting human eye-tracking data from text by combining low-level textual features with higher-level linguistic and complexity metrics. Their neural network model achieved a strong second-place finish in the CMCL 2021 Shared Task, demonstrating the effectiveness of this multi-faceted feature engineering strategy.
What Problem Does This Paper Address?
Predicting human eye-tracking behavior (like where someone looks and for how long) based on the text they are reading is a challenging task with implications for understanding reading processes, developing better human-computer interfaces, and improving accessibility. While previous work has explored various features, this paper tackles the problem of effectively integrating diverse types of information – from simple word properties to complex linguistic structures – into a single predictive model. The CMCL 2021 Shared Task specifically aimed to compare different methods for predicting five key eye-tracking variables (nFix, FFD, TRT, GPT, fixProp) averaged across multiple human subjects, given only the input text.
Methods
The authors developed a system called TALEP, which relies on a two-step process: feature generation and feature combination.
Feature Generation: They extracted a comprehensive set of features, categorized into four groups:
- Raw Textual Features: Basic properties of words, such as length, prefixes, suffixes, and word/sentence IDs.
- Frequencies: Word and bigram frequencies derived from a large language modeling corpus (Google's One Billion Words).
- Linguistic Features: Annotations generated automatically using their MACAON software, including part-of-speech (POS) tags, morphological tags, and syntactic dependencies.
- Complexity Metrics: Measures derived from the parsing process, such as stack size, distance between stack elements, and entropy of different tagging stages (tokenizer, POS tagger, dependency parser).
Feature Combination: A neural network architecture was employed to combine these features. This network consists of:
- Input Layer: Encodes features for a sliding window of three words centered on the current word. Numeric features are used directly, while discrete features (like POS tags) are mapped to learned embeddings. Bi-LSTMs process sequences of these embeddings, and a dropout layer is used for regularization.
- Multilayer Perceptron (MLP): Two linear layers with ReLU activation compress the input representation.
- Decision Layer: Five separate linear layers, each predicting one of the target eye-tracking variables.
Datasets
The MACAON model for linguistic feature extraction was trained on a concatenation of several English corpora: GUM, EWT, LinES, and ParTUT from Universal Dependencies. The shared task data itself, the ZuCo corpus, was used for training and evaluation. The authors note that their system's output was aligned with the shared task corpus using Levenshtein distances due to differences in tokenization.
Key Insights and Ideas
- Synergy of Feature Types: The paper strongly suggests that combining low-level textual features with higher-level linguistic and complexity metrics is crucial for accurate eye-tracking prediction. No single category of features was sufficient on its own.
- Complexity Metrics' Limited Impact: Surprisingly, many of the complexity metrics, particularly entropy-based ones, did not significantly improve performance beyond basic features. This contrasts with some prior work using mixed-effect models, suggesting these metrics might not be as powerful in a neural network context for this specific task.
- Stack-Based Complexity Matters: While general complexity metrics were less impactful, measures related to the syntactic parser's stack (size and distance between elements) did show some utility.
- Non-Linear Combination is Key: The authors found that their non-linear neural network approach significantly outperformed linear regressions, highlighting the importance of the model architecture in effectively leveraging the combined features.
Key Results
The TALEP system achieved a Mean Absolute Error (MAE) of 3.73 on the test data, securing second place out of 13 competing teams in the CMCL 2021 Shared Task.
- Combining "Raw Textual Features" and "Frequencies" alone yielded a strong MAE of 3.78.
- Adding "Linguistic Features" provided a small but noticeable improvement, reducing MAE by 0.04.
- The authors' neural network model achieved an MAE that was 1.23 points lower than that of linear regressions, demonstrating the benefit of their non-linear combination strategy.
Limitations and Future Work
- Complexity Metric Performance: The authors acknowledge the unexpected underperformance of most complexity metrics and suggest that while they might be significant in other modeling approaches (like mixed-effect models), they may not be powerful enough to boost the performance of their neural network.
- Feature Engineering Effort: The paper notes that the ordering of feature groups by the effort required to obtain them was a consideration. Future work could explore more automated or efficient ways to generate and select features.
- Hyperparameter Tuning: While they performed hyperparameter tuning, the decision to use the entire training material for the final model (instead of a train/dev split) led to a slightly worse MAE (3.83 vs. 3.73), indicating that careful validation splits are still important.
The paper implicitly suggests that further exploration into which specific linguistic features and complexity measures are most beneficial, and how they can be best integrated, remains a promising avenue for future research.
Summary updated