TL;DR
This paper introduces a novel approach to unsupervised dependency parsing by incorporating richer contextual information (valence and lexical features) and employing a smoothing technique inspired by supervised methods. This allows their model to achieve state-of-the-art results, significantly improving upon previous unsupervised methods.
What Problem Does This Paper Address?
Traditional unsupervised grammar induction models have often been simplified to manage data sparsity and computational tractability. While these models can learn basic syntactic structures, they struggle to capture the nuances of language. Specifically, existing unsupervised dependency parsing models, like the Dependency Model with Valence (DMV), don't fully leverage information about argument categories or lexical specifics. This paper aims to bridge this gap by developing more sophisticated unsupervised models that can learn richer syntactic representations, leading to improved parsing accuracy, especially in low-resource scenarios where annotated data is scarce.
Methods
The authors extend the Dependency Model with Valence (DMV) by introducing two new models: Extended Valence Grammar (EVG) and Lexicalized Extended Valence Grammar (L-EVG).
- Extended Valence Grammar (EVG): EVG distinguishes between the distribution of arguments nearest to the head and those farther away. This is achieved by modifying the grammar rules to allow for different distributions based on argument position, capturing a more nuanced understanding of valence.
- Lexicalized Extended Valence Grammar (L-EVG): L-EVG further enhances EVG by incorporating lexical information. This means that the model conditions argument distributions not only on the head's part-of-speech but also on the head word itself, enabling it to capture subcategorization phenomena.
- Smoothing: To combat the data sparsity introduced by these richer models, the authors adapt linear interpolation, a common smoothing technique from supervised parsing, to an unsupervised setting. They achieve this by creating a tied PCFG that allows for "backoff" distributions, effectively combining models with different levels of detail (e.g., DMV, EVG, L-EVG) and leveraging Bayesian priors. This smoothing is implemented using Variational Bayes (VB).
- Split-head Bilexical CFGs: The models are framed within the split-head bilexical CFG framework, which allows for efficient parsing using algorithms like Eisner and Satta (1999).
Datasets
The models were trained and evaluated on the Penn Treebank WSJ corpus. Specifically, they used sections 2-21 for training, section 22 as a held-out development corpus, and section 23 for evaluation. Sentences longer than 10 words (after punctuation removal) were ignored, a variant referred to as WSJ10.
Key Insights and Ideas
- Richer Contexts Improve Unsupervised Parsing: Incorporating more detailed syntactic information, such as argument position (EVG) and lexical head (L-EVG), significantly boosts the performance of unsupervised dependency parsers.
- Unsupervised Smoothing is Effective: Adapting supervised smoothing techniques like linear interpolation to an unsupervised context is a viable and powerful strategy for mitigating data sparsity in complex models.
- Combining Models is Beneficial: Mixing grammars with different levels of detail (e.g., DMV, EVG, L-EVG) through smoothing leads to state-of-the-art results, suggesting that simpler models can provide valuable backoff information for more complex ones.
- Lexicalization Matters: Even in unsupervised settings, lexical information plays a crucial role in achieving higher accuracy, enabling the model to capture finer-grained syntactic distinctions.
Key Results
The paper reports state-of-the-art results on the WSJ10 dataset.
- The L-EVG smoothed model achieved the highest directed accuracy of 68.8%, a significant improvement over previous unsupervised methods.
- Adding smoothing to DMV (DMV smoothed) improved accuracy to 61.2%, surpassing even the logistic normal prior.
- EVG smoothed-skip-head achieved 65.0% accuracy, demonstrating the benefit of distinguishing argument positions.
- The authors note that randomized initialization generally outperformed the harmonic initializer for DMV.
Limitations and Future Work
- Data Sparsity for Lexicalization: While L-EVG shows promise, the authors acknowledge that successfully using lexical information more deeply (e.g., conditioning argument words and valence on the lexical head) might require much larger datasets.
- Future Directions:
- Explore using lexical information more deeply by conditioning argument words and valence on the lexical head.
- Apply the smoothing technique to other models, such as Hidden Markov Models (HMMs).
- Learn part-of-speech tags directly from text within the dependency model, rather than relying on gold-standard tags.
Summary updated