TL;DR
This paper investigates whether combining graph-based and transition-based dependency parsers still offers benefits in the era of deep contextualized word representations like ELMo and BERT. They find that these modern representations significantly reduce the diversity between the two parser types, diminishing the gains from traditional integration methods. However, they also identify specific treebank characteristics where graph-based parsers maintain an advantage, and show that multi-task learning can still be effective for improving transition-based parsers.
What Problem Does This Paper Address?
Dependency parsing, the task of analyzing the grammatical structure of a sentence by identifying relationships between words, has traditionally been dominated by two main approaches: graph-based and transition-based parsers. These paradigms historically had complementary strengths and weaknesses, making their combination a common strategy to improve accuracy. Graph-based parsers excel at global feature optimization but can be computationally expensive, while transition-based parsers are faster and more efficient with local features but may struggle with global dependencies.
However, the advent of deep contextualized word representations (like ELMo and BERT) has revolutionized NLP. These representations encode rich semantic and syntactic information, significantly boosting the performance of both graph-based and transition-based parsers. This paper asks a crucial question: do these powerful new representations make the two parser types so similar that combining them no longer yields significant improvements, or are there still exploitable differences?
Methods
The authors re-implement the Kiperwasser and Goldberg (K&G) transition-based and graph-based parsers, incorporating deep contextualized word embeddings (specifically ELMo). They then explore two primary integration methods:
- Blending: This is a parsing-time integration method where multiple parsers' outputs are combined to form a new graph, and a graph-based decoder finds the best tree. They use a variant where multiple instances of the same parser type (e.g., multiple graph-based parsers trained with different random seeds) are blended.
- Feature-based Stacking: This method involves training one parser (level-1) to use the output of another parser (level-0) as features. They experiment with stacking transition-based on graph-based and vice-versa.
Additionally, they use Multi-Task Learning (MTL) to share BiLSTM representations between the two parsers, allowing them to learn from each other's intermediate representations without affecting parsing speed.
Datasets
The experiments are conducted on a selection of thirteen treebanks from the Universal Dependencies v2.4 dataset, chosen for their variation in language family, domain, and the proportion of non-projective arcs. These include Arabic, Basque, Chinese, English, Finnish, Hebrew, Hindi, Italian, Japanese, Korean, Russian, Swedish, and Turkish.
Key Insights and Ideas
- Deep Contextualized Representations Reduce Diversity: The most significant finding is that modern representations like BiLSTMs and ELMo make the feature sets of graph-based and transition-based parsers remarkably similar. The diversity that previously made integration beneficial is reduced to the level of differences arising from random seeds.
- Graph-Based Parsers Still Hold an Edge: Despite the convergence, graph-based parsers, especially when leveraging globally-trained BiLSTMs, consistently outperform transition-based parsers, particularly on treebanks with a high ratio of right-headed dependencies.
- Integration Gains Diminish: Traditional integration methods like blending and stacking show minimal or no significant improvement when deep contextualized representations are used. In some cases, performance even slightly drops.
- MTL Offers a Sweet Spot: Multi-task learning, by sharing intermediate representations, proves to be an effective way to improve transition-based parsers, especially on challenging treebanks, without increasing parsing time. This suggests that MTL can act as a form of regularization.
- Treebank-Specific Differences Matter: The paper highlights that the advantage of graph-based parsers is not uniform across all treebanks. Treebanks with a high proportion of non-projective arcs or right-headed dependencies show a more pronounced gap, where integration methods struggle to provide benefits.
Key Results
- When using BiLSTMs and ELMo, the performance difference between graph-based and transition-based parsers is reduced below the level of differences from random seeds.
- Blending multiple graph-based parsers yields the best results for integration, but the gains are modest when deep contextualized representations are used.
- Feature-based stacking shows little to no improvement over single parsers when deep contextualized representations are employed.
- Multi-task learning improves the transition-based parser's performance, particularly on treebanks with many right-headed dependencies, by an average of 0.42 LAS.
- Graph-based parsers consistently achieve higher LAS scores across most treebanks, with an average advantage over transition-based parsers.
Limitations and Future Work
The authors acknowledge that their study focuses on specific implementations of graph-based and transition-based parsers (K&G) and a particular set of integration methods. They suggest that exploring other advanced mechanisms or integration techniques might yield different results.
Future work could investigate:
- Whether the error profiles of transition-based parsers become even less distinguishable from graph-based outputs with the introduction of BiLSTMs and exact search algorithms.
- The impact of other deep contextualized embeddings like BERT on parser integration.
- More sophisticated multi-task learning strategies, such as sharing representations at different layers or incorporating task-specific weights.
Summary updated