TL;DR
This paper introduces RobertNLP, a surprisingly simple yet highly effective system for parsing Enhanced Universal Dependencies (EUD) in English. By jointly predicting all dependency types using a biaffine classifier on top of RoBERTa embeddings and employing a clever de-lexicalization strategy, RobertNLP achieved first place in the IWPT 2020 Shared Task, outperforming more complex pipeline approaches.
What Problem Does This Paper Address?
Universal Dependencies (UD) is a framework for annotating syntactic structure across languages. Enhanced UD (EUD) extends basic UD trees by adding more explicit dependencies, particularly for phenomena like coordination and relative clauses, aiming to better support shallow natural language understanding tasks. However, EUD structures are not always trees, making parsing them more challenging. Traditional approaches often involve a pipeline: first parsing basic UD trees, then adding enhanced relations. This can propagate errors from the first stage to the second. The IWPT 2020 Shared Task specifically focused on parsing EUD, presenting a challenge to develop robust and accurate systems for this more complex annotation scheme.
Methods
RobertNLP employs a unified, end-to-end approach to EUD parsing, treating all dependency types equally. The core of the system is a biaffine classifier, similar to those used in basic UD parsing (Dozat and Manning, 2018). This classifier takes contextualized word embeddings from RoBERTa (Liu et al., 2019b) as input and predicts the most likely dependency label (or absence of a relation, denoted by Ø) for every pair of tokens in a sentence.
A key innovation is the de-lexicalization of dependency labels. EUD has a very large number of labels due to lexical information (e.g., obl:instead_of). To combat sparsity, the system strips this lexical information during training, replacing it with placeholders (e.g., obl:[case]). This significantly reduces the label vocabulary. At prediction time, a post-processing step re-lexicalizes these placeholder labels by retrieving the relevant lexical information from the predicted dependency graph, often using rules based on dependent tokens or conjunctions.
The system also incorporates heuristics to ensure structural graph constraints, such as each token having at least one head and being reachable from a root. For the rare cases where these constraints are violated, a fallback mechanism uses an external UD parser (UDify) to generate candidate links.
Datasets
The primary dataset used for training and validation is the English EWT (Enhanced Universal Dependencies) corpus. For evaluation, the system was tested on the IWPT 2020 Shared Task blind test data. Additional experiments explored the impact of including other English UD treebanks (GUM, LinES, and ParTUT) in the training data.
Key Insights and Ideas
- Unified Parsing is Powerful: Treating basic and enhanced dependencies jointly, rather than in a pipeline, proves more effective and simpler. This avoids error propagation from a basic UD parser.
- De-lexicalization is Crucial for Sparsity: The sheer number of EUD labels is a major hurdle. Stripping lexical information and then re-attaching it later is a surprisingly effective way to manage this complexity and improve performance.
- RoBERTa's Advantage: Using RoBERTa-large embeddings, especially those that dropped the "next sentence prediction" objective, led to superior performance compared to BERT. This highlights the importance of pre-training strategies for downstream NLP tasks.
- Structural Constraints are Manageable: While EUD graphs aren't always trees, simple heuristics and a fallback mechanism are sufficient to ensure structural validity for the vast majority of sentences.
Key Results
RobertNLP achieved first place in the IWPT 2020 Shared Task on English EUD parsing, with an ELAS F1 score of 88.94%. This was a significant margin (over 1.5 points) ahead of the second-ranking submission. This performance surpassed a pipeline approach using the state-of-the-art UDify parser followed by a rule-based enhancer, which achieved an F1 score of 85.67%. The system also demonstrated high accuracy on individual label types, performing particularly well on common functional relations like det and case, and even showing promising results on enhanced-specific relations like nsubj:xsubj.
Limitations and Future Work
The authors acknowledge that their system is currently English-specific, primarily due to the language-specific handling of lexicalized relations. They believe adaptation to other languages should be feasible with moderate manual effort. They also note that while their system is robust, it is sensitive to errors in automatic segmentation and tokenization, suggesting that training on such data or improving segmentation accuracy could further boost performance. Future work could explore multi-task learning with basic UD tree parsing to potentially improve robustness and eliminate the need for external parsers for graph validation. The performance on challenging label types like punct, appos, parataxis, and list also presents avenues for further research.
Summary updated