RobertNLP at the IWPT 2021 Shared Task: Simple Enhanced UD Parsing for 17 Languages

Stefan Grünewald, Frederik Tobias Oertel, Annemarie FriedrichACL 2021

TL;DR

This paper presents RobertNLP's system for the IWPT 2021 Shared Task on Enhanced Universal Dependencies (UD) parsing. Their approach leverages a multilingual transformer (XLM-R) and a biaffine classifier, achieving a strong 3rd place finish by directly parsing into enhanced UD graphs and employing a novel de-lexicalization strategy for dependency labels.

What Problem Does This Paper Address?

Traditional Universal Dependencies (UD) represent sentence structure as trees. However, linguistic phenomena like coordination, raising, and relative clauses are better captured by "enhanced" UD, which allows for more complex graph structures beyond simple trees. This extension introduces significant challenges for parsing, as it moves away from the tree constraint and requires predicting not just the grammatical relation but also potentially lexicalized information within those relations. The IWPT 2021 Shared Task specifically focused on parsing raw text into these enhanced UD graphs across 17 languages, a task that demands robust and accurate methods for handling complex linguistic structures and a wide range of languages.

Methods

The RobertNLP system builds upon their previous English-only enhanced UD parser. Key methodological components include:

  • Multilingual Transformer Embeddings: They utilize XLM-R (Conneau et al., 2020), a powerful multilingual transformer model, to generate contextualized word embeddings for each token. The model is fine-tuned during training.
  • Unfactorized Biaffine Classifier: The core of their parser is a biaffine classifier (Dozat and Manning, 2018) that directly predicts dependency labels (including an "absence of dependency" label) for every pair of tokens. This "unfactorized" approach predicts the label and its existence simultaneously, which has been shown to be effective for dependency parsing.
  • De-lexicalization of Dependency Labels: To combat the sparsity issue caused by a large number of lexicalized dependency labels (e.g., obl:instead_of), they strip lexical information during training, replacing it with placeholders (e.g., obl:[case]). This significantly reduces the label vocabulary.
  • Hybrid Re-lexicalization: At prediction time, they employ a two-step process to restore the lexical information. First, a rule-based heuristic retrieves lexical material from the predicted dependency graph. Second, a machine-learning classifier (RandomForest) acts as a "label transducer" to correct potential errors and refine the re-lexicalized labels.
  • Model Ensembling: For each language, they train multiple models with different initializations and ensemble their predictions by averaging their output probabilities. This is done to improve robustness and accuracy.

Datasets

The paper uses data from the IWPT 2021 Shared Task. This includes training and development data for 17 languages. The specific treebanks used for each language are detailed in Table 4, which shows ensemble compositions for languages with multiple training treebanks. The evaluation is performed on a blind test set.

Key Insights and Ideas

  • Direct Parsing to Enhanced UD: Instead of relying on first parsing to basic UD trees and then converting them, the system directly predicts enhanced UD graphs from raw text. This simplifies the pipeline and potentially allows for more direct optimization.
  • De-lexicalization as a Sparsity Solution: The strategy of stripping lexical information from dependency labels during training is crucial for handling the vast number of labels in enhanced UD, especially for morphologically rich languages. This makes training more feasible and robust.
  • Hybrid Re-lexicalization is Effective: The combination of a rule-based heuristic and a machine-learning transducer for re-lexicalizing labels proves effective, achieving high accuracy on many languages, though it struggles with lemmatized gold data.
  • Ensembling Boosts Performance: Model ensembling, particularly when combining models trained on different treebanks for the same language, provides a noticeable improvement in accuracy, highlighting its benefit for robustness.

Key Results

The RobertNLP system achieved a strong 3rd place in the IWPT 2021 Shared Task, with an average ELAS F1 score of 86.97% across 17 languages. This performance was more than 3 points above the median submission.

  • The system performed exceptionally well on languages like Bulgarian and Italian (over 93% ELAS F1).
  • The de-lexicalization strategy significantly reduced label counts (e.g., to 59 for Arabic).
  • The rule-based re-lexicalization heuristic alone achieved high accuracy (over 98%) on languages like English and Estonian, but struggled with Tamil and Arabic, where lemmatization differences were more pronounced.
  • Model ensembling provided a moderate average performance increase of 0.27 ELAS F1 points, with Tamil showing the largest improvement (+1.09).

Limitations and Future Work

  • Low-Resource Performance (Tamil): The system's performance on Tamil was significantly lower (around 59% ELAS F1) due to its extremely low-resource nature (only 400 training sentences). The authors suggest that adaptations to the framework are necessary for such scenarios.
  • Re-lexicalization Accuracy: While the hybrid re-lexicalization strategy is effective, the paper notes that the heuristic can only retrieve word forms directly from the raw sentence, whereas gold labels are often lemmatized. This mismatch limits accuracy for some languages.
  • Future Work: The authors suggest exploring a purely rule-based system combined with a high-accuracy lemmatizer as a future avenue. They also aim to address the weakness in low-resource settings while maintaining the parser's core architecture.

Summary updated

BibTeX
@inproceedings{grunewald2021robertnlp,
  author = {Stefan Grünewald and Frederik Tobias Oertel and Annemarie Friedrich},
  title = {{RobertNLP at the IWPT 2021 Shared Task: Simple Enhanced UD Parsing for 17 Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure