TL;DR
This paper investigates how different types of uncertainty, specifically surprisal and entropy, influence human sentence processing. Using a self-paced reading experiment and a probabilistic context-free grammar (PCFG), they found that total entropy, but not single-step entropy, significantly predicted reading times, suggesting that a comprehensive model of sentence comprehension must account for both surprisal and broader measures of uncertainty.
What Problem Does This Paper Address?
Understanding how humans process language is a central goal in cognitive science and linguistics. A key aspect of this is how we deal with uncertainty. We know that predictable words are processed faster than unpredictable ones, a phenomenon often explained by "surprisal" – the negative log probability of a word given its context. However, the role of how the uncertainty is distributed among potential upcoming words (i.e., entropy) has been less clear. This paper explores four hypotheses about how different types of uncertainty might affect processing difficulty:
- Surprisal: Processing load is proportional to the surprisal of a word.
- Entropy Reduction: Processing is costly when uncertainty is resolved (moving from high to low entropy).
- Competition: Processing is costly when multiple interpretations are competing.
- Commitment: Processing is costly when a reader commits to a high-probability prediction.
The authors aim to disentangle these hypotheses by manipulating uncertainty in controlled experimental settings and analyzing reading times.
Methods
The researchers designed a self-paced reading experiment using a 2x2x2 factorial design. They manipulated:
- Surprisal of a sentential complement (SC): How surprising the complement clause is given the verb.
- Entropy of the verb's subcategorization distribution: How many different syntactic structures (complements) a verb can take, and how probable each is.
- Presence or absence of the complementizer "that": This creates ambiguity between a direct object and an embedded subject.
They also constructed a probabilistic context-free grammar (PCFG) from the Penn Treebank, augmented with lexically specific rules for verb subcategorization preferences. This PCFG was used to derive word-by-word estimates of surprisal, single-step entropy, and total entropy for the experimental sentences. Statistical analysis involved regressing log reading times against these predictors using linear mixed-effects models.
Datasets
The study used verbs selected from the Gahl et al. (2004) subcategorization frequency database. Verb frequencies were matched to the SUBTLEX-US corpus (Brysbaert and New, 2009). The PCFG was induced from the Penn Treebank (Marcus et al., 1993). No new datasets were introduced.
Key Insights and Ideas
- Surprisal is not the whole story: While surprisal is a significant predictor of reading times, it doesn't fully explain processing difficulty.
- Total entropy matters: The uncertainty over all possible complete parses of a sentence (total entropy) was a significant predictor of reading times, suggesting that the broader linguistic context and potential future interpretations play a crucial role.
- Single-step entropy is less informative: The uncertainty over just the next syntactic step (single-step entropy) did not significantly predict reading times. This implies that the parser's lookahead or its consideration of immediate next steps is less critical than its understanding of the overall sentence structure.
- Ambiguity vs. SC entropy: In ambiguous sentences, the embedded subject region showed increased reading times. This aligns with the competition hypothesis, where multiple parses are maintained. However, the total entropy was higher in unambiguous sentences (where the SC is certain), suggesting that total entropy might be a better measure than simply the presence of ambiguity.
- PCFG-derived measures are powerful: Using a PCFG to derive linguistic measures like surprisal and entropy proved effective in predicting reading times, offering a more principled way to quantify these theoretical constructs.
Key Results
- Surprisal: PCFG-based surprisal was a significant predictor of reading times on the disambiguating word in ambiguous sentences.
- Subcategorization Frame Entropy: This measure (single-step entropy) did not have a significant effect on reading times in any region of the sentence.
- Total Entropy: Total entropy was a significant predictor of reading times on the verb. Specifically, reading times were higher when post-verb entropy was lower, which supports the entropy reduction and commitment hypotheses, and is opposite to the competition hypothesis.
- Ambiguity: Reading times on the embedded subject were higher in ambiguous sentences compared to unambiguous ones.
Limitations and Future Work
- Representational Decisions: The authors acknowledge that the calculation of entropy can be sensitive to representational choices in the grammar (e.g., how nodes are annotated). Future work should explore the impact of these decisions.
- Lexicalization: Using a lexicalized grammar (as advocated by some researchers) would make PCFG inversion computationally impractical due to the large number of nonterminal symbols. The current approach uses a mostly unlexicalized grammar with lexically specific rules.
- Naturalistic Corpora: The current study used controlled experimental materials. Future work should investigate the effect of entropy in naturalistic reading time corpora, which would provide a more direct test of the models in real-world language use.
- Four-Word Lookahead: The authors note that simple lookahead mechanisms (like four-word lookahead) might not fully capture the complex internal structure of syntactic constituents, which is important for understanding entropy effects.
Summary updated