TL;DR
This paper demonstrates that hierarchical syntactic structure significantly improves predictions of human reading times, even when accounting for sophisticated n-gram language models. By using a more robust n-gram baseline and carefully analyzing eye-tracking data, the researchers show that linguistic structure provides crucial information beyond simple word sequence probabilities.
What Problem Does This Paper Address?
A long-standing debate in psycholinguistics concerns whether humans utilize complex hierarchical syntactic information when processing language, or if simpler sequential (n-gram) models are sufficient to explain reading behavior. Previous studies have yielded mixed results, with some suggesting that hierarchical measures don't significantly improve reading time predictions over strong n-gram baselines. This paper aims to resolve this by employing a more powerful n-gram baseline and rigorously testing the predictive power of hierarchical syntactic measures. The core question is: does the brain build and use a complex "tree" of grammatical relationships, or does it primarily rely on predicting the next word based on the last few?
Methods
The authors employ linear mixed-effects models to predict reading times (specifically, first-pass and go-past durations) from eye-tracking data.
- Improved N-gram Baseline: Instead of simple bigrams or n-grams from the end of a region, they construct an interpolated 5-gram baseline using a massive corpus (2.96 billion words). Crucially, they also explore "cumulative" n-gram factors, which sum probabilities over an entire eye-tracking region, rather than just the last word.
- Hierarchical Syntax Measures: They use "PCFG surprisal" as a measure of hierarchical syntactic processing. This quantifies how surprising a word is given the current syntactic structure being built. They test two versions:
- Penn Treebank (PTB): A standard, manually annotated grammar.
- Generalized Categorial Grammar (GCG): A linguistically-motivated grammar that explicitly models long-distance dependencies.
- Ablative Testing: They systematically add and remove predictors (n-gram factors, syntactic factors) to assess the unique contribution of each.
Datasets
- Dundee Corpus: This is the primary dataset used for evaluation. It consists of eye-tracking data from 10 subjects reading 2388 sentences from The Independent newspaper. Preprocessing involved filtering out initial/final fixations and long saccades to avoid confounds.
- Gigaword 4.0 Corpus: Used to compute the interpolated 5-gram probabilities for the n-gram baseline. This corpus contains approximately 2.96 billion words from around 4 million English newswire documents.
Key Insights and Ideas
- Cumulative N-grams Matter: Simply using the last n-gram in a region is less effective than aggregating n-gram statistics over the entire region. This suggests that readers process the sequential predictability of a whole chunk of text, not just the immediate preceding words.
- Hierarchical Syntax is Not Redundant: Even with a very strong, cumulative n-gram baseline, hierarchical syntactic measures (PCFG surprisal) still provide significant, orthogonal improvements in predicting reading times. This directly challenges the idea that n-grams can fully account for syntactic effects.
- GCG Captures Something PTB Misses: The Generalized Categorial Grammar (GCG), which is designed to handle long-distance dependencies, offers unique predictive power beyond the standard PTB grammar. This suggests that readers do indeed use information about distant grammatical relationships.
- Local Structure is Key, But Not Everything: While cumulative n-grams and PTB surprisal (which captures local syntactic structure) are important, the GCG model's ability to capture long-distance dependencies indicates that these are also utilized by readers.
Key Results
- An interpolated 5-gram baseline, especially when using cumulative probabilities over regions, significantly improves reading time prediction compared to simpler n-gram models.
- PCFG surprisal, derived from both PTB and GCG grammars, significantly improves reading time predictions over the strong n-gram baseline.
- When both PTB and GCG surprisal measures are included, they both contribute significantly, indicating they capture different aspects of hierarchical processing.
- The GCG model, which explicitly models long-distance dependencies, provides a better fit than the PTB model, suggesting that readers utilize these longer-range syntactic relationships.
- Cumulative PCFG surprisal (summing surprisal over a region) did not significantly improve predictions when a cumulative n-gram factor was already present, suggesting that the benefits of cumulative processing might be more strongly tied to sequential frequency than hierarchical structure in this specific context.
Limitations and Future Work
- Cumulative Surprisal: The finding that cumulative PCFG surprisal didn't add value when cumulative n-grams were present is noted as interesting, suggesting that non-local hierarchical structure might not influence reading times as much as local structure, or that its effect is already captured by other factors.
- GCG Annotation Errors: The authors acknowledge that potential errors in the automatically generated GCG grammar might affect its estimates, and that the manually annotated PTB grammar might be credited for correct predictions where GCG errs.
- Future Work: The authors suggest investigating the distinction between lexical, syntactic, and total surprisal in future work, especially as sequential baselines improve. They also propose exploring the use of cumulative n-grams to predict spillover effects and further investigating the role of non-local dependencies.
Summary updated