TL;DR
This paper investigates how well two theories of language processing difficulty—Dependency Locality Theory (DLT) and Surprisal Theory—predict human reading times. By using modern neural language models to estimate surprisal and comparing it with DLT integration costs derived from a hand-parsed treebank, the study finds that while neural surprisal is a strong predictor overall, DLT is particularly important for predicting difficulty with nouns. This suggests that a comprehensive model of language processing needs to integrate both expectation-based (surprisal) and memory-based (DLT) factors.
What Problem Does This Paper Address?
Understanding why some sentences are harder to process than others is a fundamental question in psycholinguistics and computational linguistics. Two prominent theories attempt to explain this: Dependency Locality Theory (DLT) posits that processing difficulty increases with the distance between syntactically related words, while Surprisal Theory suggests that difficulty is inversely related to the probability of a word appearing in a given context. Previous work has compared these theories, but often relied on older methods for calculating surprisal (like PCFGs or n-grams) which are less sophisticated than current neural language models. This paper aims to bridge this gap by using state-of-the-art neural networks to estimate surprisal and comparing these estimates, along with DLT integration costs, to actual human reading times. The goal is to determine which theory, or combination of theories, best predicts processing difficulty in a more nuanced way.
Methods
The study employs a methodology similar to prior work in empirically testing processing theories, using reading time data as a proxy for processing difficulty.
- Reading Time Data: The researchers utilized the Dundee Corpus, a large dataset of eye-tracking data from ten English-speaking participants reading newspaper articles. First-pass gaze durations were used as the measure of reading time.
- Integration Cost (IC) Calculation: DLT integration costs were calculated using the Dundee Treebank, a hand-parsed dataset. This manual parsing is considered more accurate than automatic parsing used in previous studies. The researchers also accounted for syntactic traces by adding costs based on specific dependency relations.
- Surprisal Models: Two neural language models were used to estimate surprisal:
- A Recurrent Neural Network (RNN) model trained on a large English Wikipedia corpus.
-
The GPT-2 Transformer model (1.5 billion parameter version) trained on the English WebText corpus. These models are chosen because recent research suggests they can learn syntactic structures, making them suitable for estimating word probabilities in context.
-
Statistical Analysis: Mixed-effects models were used to analyze the relationship between reading times, IC, and surprisal, accounting for individual participant and item variability. Spillover effects (where the difficulty of a preceding word affects the current word's reading time) were addressed by including the previous word's surprisal or IC in the model. Generalized Additive Models (GAMs) were also used to visualize the relationship between raw surprisal values and reading times.
Datasets
- Dundee Corpus: Used for eye-tracking data (reading times). This corpus consists of English text from the Independent newspaper, with data from ten participants.
- Dundee Treebank: Used for calculating Dependency Locality Theory (DLT) integration costs. This is a hand-parsed treebank derived from the Dundee Corpus.
- English Wikipedia Corpus: Used to train the RNN surprisal model.
- English WebText Corpus: Used to train the GPT-2 Transformer surprisal model.
No new benchmarks were introduced; the study leverages existing corpora and treebanks.
Key Insights and Ideas
- Neural Surprisal is a Strong General Predictor: Modern neural language models provide surprisal estimates that are significantly correlated with reading times across the entire dataset, indicating their power in predicting general processing difficulty.
- DLT's Nuance for Nouns: While DLT integration cost showed a negative relationship with reading times overall (contrary to its theoretical predictions), it emerged as a strong positive predictor when the analysis was restricted to nouns. This highlights that DLT might be more effective for specific word categories.
- The Need for a Hybrid Model: The study strongly suggests that neither DLT nor Surprisal Theory alone can fully capture processing difficulty. A comprehensive model likely needs to integrate both expectation-based (surprisal) and memory-based (DLT) mechanisms.
- Neural Models Outperform Older Methods: The use of neural LMs for surprisal estimation is shown to be more effective than older methods like PCFGs, aligning with findings that neural models better capture human-like language processing.
- Minimal Correlation Between Surprisal and IC: Similar to previous work, the study found only a weak correlation between surprisal and integration cost, reinforcing the idea that they capture different aspects of processing difficulty.
Key Results
- Overall Data:
- Neural surprisal (from both RNN and GPT-2) had a significant positive coefficient, meaning higher surprisal correlates with longer reading times.
- Integration Cost (IC) had a significant negative coefficient, indicating that higher IC correlates with shorter reading times.
- Nouns Only:
- When analyzing only nouns, IC showed a significant positive coefficient, aligning with DLT's predictions for specific word types.
- Surprisal remained a significant positive predictor for nouns.
- Spillover Effects: Surprisal showed significant spillover effects, while IC did not.
- Correlations: There was a moderate to high correlation between the surprisal values from the RNN and GPT-2 models, but minimal correlation between surprisal and IC.
Limitations and Future Work
- Limitations:
- The Dundee Corpus, while large, consists mostly of common syntactic constructions, which might not fully represent the complexity of language processing in all contexts. This could lead to a less generalized picture of processing difficulty.
- The negative coefficient for IC on the overall dataset is attributed to DLT assigning zero cost to non-discourse referents, which might mask its true predictive power.
- Future Work:
- The authors suggest that future work should aim to combine expectation-based models (like Surprisal Theory) with memory-based theories (like DLT) to create a more robust, broad-coverage model of sentence processing.
- Exploring modified forms of IC that might better predict non-discourse referent processing difficulties is also suggested.
- Investigating how these theories account for processing differences in ambiguities is an area for further research.
Summary updated