TL;DR
This paper explores how to leverage eye-tracking data to improve dependency parsing, specifically when this data is only available during training and not at inference time. They propose a multi-task learning approach where gaze prediction acts as an auxiliary task, showing modest but positive gains in parsing accuracy.
What Problem Does This Paper Address?
Natural Language Processing (NLP) tasks, like dependency parsing, aim to understand the grammatical structure of sentences. Eye-tracking data, which captures how humans read, has shown promise in improving NLP models. However, a significant challenge is that eye-tracking data is often expensive and difficult to collect, meaning it's typically only available for a limited set of training data, not for real-time inference. This paper tackles the problem of effectively using this "training-only" gaze data to enhance dependency parsers without requiring gaze information during the parsing process itself.
Methods
The core of the paper's method is a multi-task learning (MTL) framework. They frame both dependency parsing and gaze prediction as sequence labeling tasks.
- Dependency Parsing as Sequence Labeling: They adapt a method that linearizes a dependency tree into a sequence of labels for each word. Each label encodes the relationship between a word and its head (e.g., dependency type and the index of the head word). This is implemented using a bidirectional Long Short-Term Memory (BiLSTM) network.
- Gaze Prediction as Auxiliary Task: Eye-tracking data is pre-processed into discrete features (e.g., total fixation duration, number of fixations, fixation probability). These gaze features are then used as targets for auxiliary prediction tasks within the same MTL framework.
- Hard-Sharing Architecture: The BiLSTM layers are shared across both the main dependency parsing task and the auxiliary gaze prediction tasks. Separate feed-forward layers with softmax are used for each task's output. This allows the model to learn a shared representation that benefits both tasks.
- Disjoint Datasets: A key aspect is their exploration of using gaze data from one corpus (Dundee) to improve parsing on a different corpus that only has dependency annotations (Penn Treebank). This addresses the scenario where gaze data is not available for the target parsing dataset.
Datasets
- Dundee Corpus: This dataset contains eye-movement recordings from ten English-speaking participants reading newspaper articles. It was used to create the Dundee Treebank, which augments the Dundee corpus with Universal Dependencies (UD) syntactic annotations. This serves as the parallel data (containing both gaze and dependency annotations).
- Penn Treebank (PTB): This is a widely used dataset for dependency parsing, containing sentences from The Wall Street Journal annotated with phrase-structure trees. It was converted to Stanford Dependencies. This dataset is used for disjoint data experiments, where it provides the dependency annotations, while gaze data comes from the Dundee corpus.
Key Insights and Ideas
- Leveraging Training-Only Gaze Data is Feasible: The paper demonstrates that even when gaze data is restricted to the training phase, it can still provide a benefit to dependency parsing performance.
- Auxiliary Tasks are Effective: Using gaze prediction as an auxiliary task within an MTL framework is a viable strategy to inject gaze information into the parser's learning process.
- Disjoint Data Transfer is Possible (with caveats): The experiments show that gaze data from one source can indeed help improve parsing on a different dataset, suggesting a degree of generalization, though gains are more modest compared to parallel data.
- Feature Grouping Matters: Treating groups of gaze features (e.g., "early" or "context" features) as multiple auxiliary tasks can lead to more consistent improvements than individual features.
Key Results
- Parallel Data (Experiment 1): Modest but positive accuracy gains were observed. For instance, using "context feats aux" improved LAS by +0.18 on the test set. The paper notes that improvements were "unstable" across different features and sets.
- Disjoint Data (Experiment 2): Gains were smaller compared to the parallel setup, but more consistent between development and test sets. For example, "mean fix dur" increased LAS by +0.17 on the dev set. Grouping features as "context feats aux" improved LAS by +0.18 on the test set.
- Baseline Performance: The baseline models (without gaze features) achieved competitive results, indicating the strength of the underlying parsing architecture.
Limitations and Future Work
- Modest Gains: The accuracy improvements are described as "modest," suggesting there is significant room for further enhancement.
- Generalization Across Treebanks: The paper acknowledges that generalization capabilities across different treebanks are an area for improvement, especially in the disjoint setup.
- Architecture Exploration: The authors suggest that exploring different architectures, potentially adapting human-attention approaches (like those by Barrett et al., 2018) for structured prediction, could lead to better utilization of gaze information.
- Speed vs. Performance: While attention mechanisms might improve performance, they could come at a cost to parsing speed, which is a trade-off to consider.
- Broader Data Types: The paper notes that their approach is not limited to eye-tracking data and could be extended to leverage other types of human-generated data (e.g., NER or chunking annotations).
Summary updated