Abstract
Sigrid Klerke, Barbara Plank. Proceedings of the Beyond Vision and LANguage: inTEgrating Real-world kNowledge (LANTERN). 2019.
This paper investigates how to best aggregate human eye-tracking data to improve NLP models for syntactic tagging. The researchers discovered that "global" views—averaging gaze behavior across many readers or measuring their variance—are far more effective than using individual, "local" reading traces. Furthermore, gaze data was found to be significantly more helpful for Part-of-Speech (POS) tagging than for phrase chunking.
Introduction
When we read, our eyes do not move smoothly across the page; they skip easy words, linger on complex ones, and backtrack when we encounter linguistic ambiguity. These "digital traces" of human cognition are a potential goldmine for training NLP models, as they signal which parts of a sentence are linguistically demanding.
However, the field has lacked a systematic understanding of how to represent this data. Should a model learn from one specific person's gaze on a specific word (a "local, token-level" view)? Or should it look at how people generally read that word across many contexts (a "global, type-level" view)? This paper tackles the "aggregation problem": finding the right balance between signal and noise in gaze data to boost performance in syntactic sequence labeling.
Methodology
The authors compared two primary dimensions of gaze representation across two low-level syntactic tasks: binary phrase chunking and Part-of-Speech (POS) tagging.
Aggregation Levels
- Local vs. Global: Local views retain individual participant traces. Global views aggregate data into a "typical reader" profile using the mean (central tendency) or variance (how much readers disagree).
- Token vs. Type: Token-level models use gaze data tied to a specific word in a specific sentence (contextualized). Type-level models use a "gaze dictionary" where a word's average gaze behavior is used as a static feature, regardless of context.
Gaze Metrics
The study focused on "Early" measures (first-pass duration) and "Late" measures (total time including regressions and backtracking). These were binned into six categorical levels (0-5) based on standard deviations from the mean.
Modeling and Datasets
Two modeling approaches were tested: Multi-task Learning (MTL), where the model predicts gaze as an auxiliary task, and Dictionary Modeling, where gaze features are concatenated to word embeddings. The study utilized two major eye-tracking corpora to ensure generalizability:
- The Dundee Corpus: 10 native English speakers reading newspaper articles (news genre).
- GECO (Ghent Eye-Tracking Corpus): 14 native English speakers reading a complete novel (fiction genre).
Key Insights
- The "Typical Reader" Wins: Aggregating data (Global view) is almost always superior to using individual traces (Local view). Individual reading patterns are too noisy; models perform better when learning from the consensus of how a "hypothetical typical reader" processes text.
- Decontextualization is Surprisingly Effective: Type-level dictionaries often outperformed token-level multi-task learning. This suggests that gaze data is most useful as a proxy for a word's inherent "difficulty" or "category" rather than its specific role in a single sentence.
- Task Specificity Matters: Gaze data is a strong signal for POS tagging (improving results in 9 out of 12 setups) but provides almost no benefit for phrase chunking. This implies that while eye movements reflect word-level identity well, they may not capture the structural boundaries required for chunking as effectively.
- Variance as a Signal: The authors introduced gaze variance (how much readers' eye movements differ on a word) as a feature. In some cases, knowing that readers disagree on a word was more informative than knowing the average time they spent on it.
Key Results
- POS Tagging: The best models (using type-level global means) achieved up to a +0.23 absolute accuracy improvement over the baseline (95.25%).
- Phrase Chunking: Results were largely negative. Most gaze-augmented models performed worse than the baseline (94.93 F1), with only one specific setup showing a marginal, non-robust improvement.
- Local View Detriment: Using un-aggregated individual traces consistently led to performance drops, confirming that raw, individual gaze data is too high-variance for these models to utilize effectively.
Conclusion
Gaze data is a valuable but sensitive resource for NLP. Its effectiveness depends heavily on the task, the corpus, and the aggregation strategy. Future research should investigate why gaze helps with content words but often confuses models when applied to function words, and explore more sophisticated ways to handle "null" gaze values (when a reader skips a word).
Summary updated