TL;DR
This paper introduces a novel neural network-based incremental parser that mimics human working memory limitations. By constraining its internal memory, the parser achieves competitive parsing accuracy while providing more human-like predictions for psycholinguistic phenomena like reading times, bridging a gap between current neural models and cognitive reality.
What Problem Does This Paper Address?
Modern neural network parsers have achieved impressive accuracy in syntactic parsing. However, many of these models, particularly those using shift-reduce architectures, employ "unbounded stack memory." This means they can store an arbitrarily long history of parsed elements, which is inconsistent with human cognitive limitations, specifically bounded working memory. This discrepancy poses a problem when using these parsers to predict human behavior, such as reading times, as the unbounded memory might artificially inflate parsing accuracy or distort the surprisal measures that are meant to reflect cognitive load. The paper aims to develop a neural parser that is both accurate and respects these human memory constraints, making it a more suitable tool for psycholinguistic modeling.
Methods
The core of this work is a memory-bounded neural incremental left-corner parser. This approach combines a left-corner transition system with a neural network architecture.
- Left-Corner Transition System: This system is chosen because it naturally has more limited stack memory requirements compared to shift-reduce parsers, especially for linguistically common structures. Stack depth only increases significantly with center-embedded clauses, which are known to be difficult for humans.
- Neural Network Architecture: The parser uses a stack-LSTM architecture, which is a type of recurrent neural network designed to handle sequential data and maintain a state. This architecture is augmented with components for processing different types of decisions (preterminal, top, bottom) and for composing the stack state.
- Bounded Stack: Crucially, the model explicitly constrains the depth of the stack memory. This is achieved through specific constraint sets for parsing, including a "basic set" for psycholinguistic measures and an "extended set" for ensuring well-formed parses. The maximum depth is set to 5 derivation fragments (10 tree nodes), reflecting human memory limits.
- Training: The model is trained in two stages: first with a cross-entropy loss objective, and then fine-tuned using the REINFORCE algorithm for sequence-level supervision.
Datasets
The paper uses the following datasets:
- Penn Treebank (Wall Street Journal sections 02-21): Used for training the neural parser. Sections 22 and 23 are used for development and testing, respectively, for linguistic accuracy evaluation.
- Natural Stories Corpus: Used for psycholinguistic evaluation, specifically for regressing surprisal predictors to reading times.
- Dundee Corpus: Used for psycholinguistic evaluation, specifically for regressing surprisal predictors to eye-tracking data (first-pass and go-past fixation durations).
Key Insights and Ideas
- Bridging the Gap: The paper successfully demonstrates that neural parsing accuracy can be maintained even when memory is bounded, suggesting that the "unboundedness" of some current neural parsers might not be strictly necessary for high performance and could even be detrimental for cognitive modeling.
- Left-Corner Advantage: The left-corner parsing strategy inherently offers a more human-like memory profile, making it a good foundation for building cognitively plausible parsers.
- Surprisal as a Predictor: The work reinforces the utility of syntactic surprisal derived from incremental parsers as a strong predictor of human reading behavior, and shows that memory-bounded surprisal is more aligned with human data.
- Model Performance: The bounded neural parser not only performs competitively in parsing accuracy but also shows superior performance in fitting human behavioral data compared to both unbounded neural parsers and previous memory-bounded statistical parsers.
Key Results
- Linguistic Accuracy: The proposed memory-bounded neural parser achieved a labeled bracketing F1 score of 90.1% on the development set and 89.5% on the test set (WSJ section 23). This is competitive with memory-unbounded parsers (e.g., Hale et al. (2018) at 91.3%) and outperforms previous memory-bounded statistical parsers like van Schijndel et al. (2013) (87.8%).
- Psycholinguistic Accuracy: In psycholinguistic evaluations using the Natural Stories Corpus and Dundee Corpus, the surprisal values derived from the bounded neural parser provided a better fit to human reading times and eye-tracking data compared to surprisal from a previous state-of-the-art memory-bounded statistical parser (van Schijndel et al., 2013). Adding the neural surprisal to models already including the statistical surprisal significantly improved the fit, while adding the statistical surprisal to models with neural surprisal did not.
Limitations and Future Work
- Slight Accuracy Drop: The paper acknowledges a slight decrease in parsing accuracy (1.2 points) compared to an unbounded neural parser (Hale et al., 2018). This is attributed to the bounded stack potentially losing some information about past parsing decisions.
- Future Work: The authors suggest that the left-corner model's bounded stack memory can also help limit the search space for unsupervised grammar acquisition, indicating a potential avenue for future research. The paper also implies that further exploration of different constraint sets for parsing could be beneficial.
Summary updated