TL;DR
This paper tackles the challenge of predicting human eye-tracking behavior during reading using advanced language models. The team fine-tuned RoBERTa, a powerful pre-trained language model, and augmented it with simple yet effective features like word length and sentence progress, achieving strong results that outperform baselines and are competitive with other shared task participants.
What Problem Does This Paper Address?
Understanding how humans process language in real-time is crucial for cognitive science and natural language processing. Eye-tracking data, such as fixation durations and locations, provides rich insights into this process, reflecting cognitive effort and difficulty. However, accurately predicting these subtle behavioral responses from text alone has been a significant challenge. Previous work has relied on hand-crafted psycholinguistic features, but the advent of large pre-trained language models (PLMs) like RoBERTa offers a new avenue for capturing complex linguistic patterns that might underlie eye movements. This paper addresses the gap of effectively leveraging PLMs for eye-tracking prediction and exploring the utility of additional, simple features in this context.
Methods
The core of the Team Ohio State's approach is transfer learning using a pre-trained language model. They fine-tuned RoBERTa (specifically, both the base and large variants) on eye-tracking data. A key challenge with using RoBERTa is its byte-pair encoding (BPE) tokenization, which results in a mismatch between the number of BPE tokens and the number of actual words in a sentence. To handle this, they used the representation of the first BPE token associated with each word as the word's representation.
Beyond the RoBERTa representations, they introduced two additional input features, motivated by prior eye-tracking research:
- Word length in characters (wlen): This feature captures the intuition that longer words tend to elicit longer fixations.
- Proportion of sentence processed (prop): This is calculated as the current word's index divided by the total number of words in the sentence. This feature aims to account for "edge effects" (e.g., at the beginning or end of a sentence).
These three features (RoBERTa representation, wlen, prop) were fed into a simple feedforward neural network (FFNN) with one hidden layer. Crucially, a separate FFNN was trained for each of the five eye-tracking features.
Datasets
The primary dataset used for training and evaluation is the Zurich Cognitive Language Processing Corpus (ZuCo 1.0 and 2.0). This corpus contains naturalistic self-paced reading data from native English speakers, along with corresponding eye-tracking features. The sentences in ZuCo are derived from the Stanford Sentiment Treebank and the Wikipedia relation extraction corpus.
For training, the authors used 720 sentences, setting aside the last 80 sentences (approximately 10%) as a held-out development (dev) set for hyperparameter tuning. The final evaluation was performed on the test set.
Key Insights and Ideas
- PLMs are powerful predictors: Fine-tuning RoBERTa significantly outperforms a simple mean baseline, demonstrating the model's ability to capture complex linguistic patterns relevant to eye movements.
- Simple features matter: Despite the power of RoBERTa, adding basic features like word length and sentence position still provides a benefit, suggesting that these intuitive linguistic properties are not fully implicitly captured by the PLM's contextual representations alone.
- Word length is particularly useful: The ablation study revealed that removing the word length feature had a more noticeable negative impact on prediction accuracy than removing the sentence proportion feature. This highlights the direct relationship between orthographic length and fixation behavior.
- Orthogonality of features: The utility of word length suggests it captures information orthogonal to what RoBERTa's BPE-based representations provide, especially given the model's strategy of using only the first token's representation for each word.
- Capacity vs. Data: The finding that the smaller RoBERTa-base model performed slightly better than RoBERTa-large on the test set suggests that with limited training data, larger models might be more prone to overfitting, and simpler models can be more robust.
Key Results
- Both RoBERTa-base and RoBERTa-large models outperformed the mean baseline by approximately 40% in terms of Mean Absolute Error (MAE) across all five eye-tracking features.
- The full model (RoBERTa-base + wlen + prop) achieved the best performance, with MAE values on the test set being:
- nFix: 3.987
- FFD: 0.682
- GPT: 2.364
- TRT: 1.540
- fixProp: 11.311
- An ablation study showed that removing the
propfeature had a negligible impact on MAE. However, removing thewlenfeature led to a slight increase in MAE for most features, indicating its contribution. The combination of bothwlenandpropdid not yield further improvements over the full model. - The RoBERTa-base model ranked 11th out of 29 submissions in the shared task.
Limitations and Future Work
- BPE Tokenization Mismatch: The paper acknowledges that using only the first BPE token's representation for each word is a simplification. Future work could explore more sophisticated methods for aggregating BPE token representations to better capture word-level semantics.
- Positional Information: While the
propfeature attempts to capture positional effects, the authors note that the Transformer architecture itself (on which RoBERTa is based) inherently uses positional encodings. A more thorough investigation would involve evaluating models trained without positional encodings to fully understand the contribution of explicit positional features. - Data Size: The limited size of the training data is a factor that might influence model capacity choices. Future work could explore how these models perform with larger datasets.
- Implicit vs. Explicit Features: The paper hypothesizes that RoBERTa implicitly captures frequency and surprisal information. Further research could explicitly test this by including these features and comparing performance.
Summary updated